[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 557154006 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003289] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008507] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022329] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027050] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029071] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031072] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] *** VALIDATE tmpfs *** [ 0.033468] *** VALIDATE proc *** [ 0.035151] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.038024] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.044922] debug: unmapping init [mem 0xffffffff91e59000-0xffffffff91e60fff] [ 0.046176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055334] rcu: Hierarchical SRCU implementation. [ 0.057418] smp: Bringing up secondary CPUs ... [ 0.058591] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.062094] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146401] node 0 deferred pages initialised in 79ms [ 0.150098] devtmpfs: initialized [ 0.151212] x86/mm: Memory block size: 128MB [ 0.153795] gcov: version magic: 0x41383552 [ 0.154536] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158140] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161337] pinctrl core: initialized pinctrl subsystem [ 0.163189] [ 0.163786] ************************************************************* [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168011] ** ** [ 0.171013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173010] ** ** [ 0.175014] ** This means that this kernel is built to expose internal ** [ 0.177011] ** IOMMU data structures, which may compromise security on ** [ 0.180013] ** your system. ** [ 0.182012] ** ** [ 0.184013] ** If you see this message and you are not debugging the ** [ 0.187015] ** kernel, report this immediately to your vendor! ** [ 0.189010] ** ** [ 0.191013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193014] ************************************************************* [ 0.196613] NET: Registered protocol family 16 [ 0.198444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207064] cpuidle: using governor menu [ 0.208540] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210389] PCI: Using configuration type 1 for base access [ 0.213134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227073] cryptd: max_cpu_qlen set to 1000 [ 0.230168] ACPI: Added _OSI(Module Device) [ 0.231011] ACPI: Added _OSI(Processor Device) [ 0.232009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237128] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241387] ACPI: Interpreter enabled [ 0.242045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243009] ACPI: Using IOAPIC for interrupt routing [ 0.244103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248360] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258890] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271402] acpiphp: Slot [2] registered [ 0.273153] acpiphp: Slot [5] registered [ 0.274112] acpiphp: Slot [6] registered [ 0.275159] acpiphp: Slot [3] registered [ 0.276073] acpiphp: Slot [4] registered [ 0.278077] acpiphp: Slot [7] registered [ 0.279068] acpiphp: Slot [8] registered [ 0.280062] acpiphp: Slot [9] registered [ 0.281106] acpiphp: Slot [10] registered [ 0.282000] acpiphp: Slot [11] registered [ 0.282000] acpiphp: Slot [12] registered [ 0.284179] acpiphp: Slot [13] registered [ 0.286093] acpiphp: Slot [14] registered [ 0.287135] acpiphp: Slot [15] registered [ 0.289174] acpiphp: Slot [16] registered [ 0.291133] acpiphp: Slot [17] registered [ 0.293120] acpiphp: Slot [18] registered [ 0.294114] acpiphp: Slot [19] registered [ 0.296119] acpiphp: Slot [20] registered [ 0.298114] acpiphp: Slot [21] registered [ 0.300122] acpiphp: Slot [22] registered [ 0.301184] acpiphp: Slot [23] registered [ 0.303117] acpiphp: Slot [24] registered [ 0.304082] acpiphp: Slot [25] registered [ 0.305093] acpiphp: Slot [26] registered [ 0.307106] acpiphp: Slot [27] registered [ 0.308102] acpiphp: Slot [28] registered [ 0.310099] acpiphp: Slot [29] registered [ 0.312106] acpiphp: Slot [30] registered [ 0.313244] acpiphp: Slot [31] registered [ 0.315060] PCI host bridge to bus 0000:00 [ 0.317020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329129] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332024] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335252] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342007] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.346056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354620] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356878] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364916] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368972] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.379020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.383015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388260] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436307] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498993] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.502527] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.505448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.508457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.510235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.515097] iommu: Default domain type: Passthrough [ 0.516008] SCSI subsystem initialized [ 0.517147] ACPI: bus type USB registered [ 0.519070] usbcore: registered new interface driver usbfs [ 0.520077] usbcore: registered new interface driver hub [ 0.522080] usbcore: registered new device driver usb [ 0.524166] pps_core: LinuxPPS API ver. 1 registered [ 0.526014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.529086] PTP clock support registered [ 0.532072] EDAC MC: Ver: 3.0.0 [ 0.533396] PCI: Using ACPI for IRQ routing [ 0.536000] NetLabel: Initializing [ 0.538016] NetLabel: domain hash size = 128 [ 0.540014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541289] NetLabel: unlabeled traffic allowed by default [ 0.544469] vgaarb: loaded [ 0.546287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.548014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.558355] clocksource: Switched to clocksource kvm-clock [ 0.658414] VFS: Disk quotas dquot_6.6.0 [ 0.659516] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.661284] *** VALIDATE ramfs *** [ 0.662156] *** VALIDATE hugetlbfs *** [ 0.663244] pnp: PnP ACPI init [ 0.664843] pnp: PnP ACPI: found 6 devices [ 0.677953] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.680224] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.681804] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.683185] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.684800] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.686498] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.688752] NET: Registered protocol family 2 [ 0.691173] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.696790] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.702114] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.707877] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.711912] TCP: Hash tables configured (established 65536 bind 65536) [ 0.714869] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.717949] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.720956] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.724062] NET: Registered protocol family 1 [ 0.726712] RPC: Registered named UNIX socket transport module. [ 0.728697] RPC: Registered udp transport module. [ 0.730342] RPC: Registered tcp transport module. [ 0.732104] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.734954] NET: Registered protocol family 44 [ 0.736453] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.738690] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.740484] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.742529] PCI: CLS 0 bytes, default 64 [ 0.744129] Unpacking initramfs... [ 2.107296] debug: unmapping init [mem 0xffff8e7ebcc64000-0xffff8e7ebffcffff] [ 2.111167] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.113275] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.115518] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.585694] Initialise system trusted keyrings [ 2.588308] Key type blacklist registered [ 2.589873] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.598708] zbud: loaded [ 2.601759] *** VALIDATE nfs *** [ 2.603111] *** VALIDATE nfs4 *** [ 2.604850] pstore: using deflate compression [ 2.607965] Platform Keyring initialized [ 2.697326] NET: Registered protocol family 38 [ 2.698712] Key type asymmetric registered [ 2.700272] Asymmetric key parser 'x509' registered [ 2.702232] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.706041] io scheduler mq-deadline registered [ 2.707621] io scheduler kyber registered [ 2.708849] io scheduler bfq registered [ 2.710233] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.712736] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.715425] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.719291] ACPI: Power Button [PWRF] [ 2.724033] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.728839] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.739361] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.767474] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.796530] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.801671] Non-volatile memory driver v1.3 [ 2.803472] Linux agpgart interface v0.103 [ 2.835309] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 2.838401] vda: detected capacity change from 0 to 74645504 [ 2.853421] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.856479] vdb: detected capacity change from 0 to 1073741824 [ 2.862755] libphy: Fixed MDIO Bus: probed [ 2.869372] usbcore: registered new interface driver usbserial_generic [ 2.872261] usbserial: USB Serial support registered for generic [ 2.875075] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.881706] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.883708] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.886821] mousedev: PS/2 mouse device common for all mice [ 2.889977] rtc_cmos 00:05: RTC can wake from S4 [ 2.892439] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.894066] rtc_cmos 00:05: registered as rtc0 [ 2.898623] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.901522] intel_pstate: CPU model not supported [ 2.904662] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.908315] hid: raw HID events driver (C) Jiri Kosina [ 2.910508] usbcore: registered new interface driver usbhid [ 2.912616] usbhid: USB HID core driver [ 2.914277] drop_monitor: Initializing network drop monitor service [ 2.914673] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.916426] Initializing XFRM netlink socket [ 2.921603] NET: Registered protocol family 10 [ 2.924553] Segment Routing with IPv6 [ 2.926318] NET: Registered protocol family 17 [ 2.928727] mpls_gso: MPLS GSO support [ 2.934897] RAS: Correctable Errors collector initialized. [ 2.936555] AVX version of gcm_enc/dec engaged. [ 2.937730] AES CTR mode by8 optimization enabled [ 3.017496] sched_clock: Marking stable (3017461699, 0)->(3897071786, -879610087) [ 3.022608] registered taskstats version 1 [ 3.024968] Loading compiled-in X.509 certificates [ 3.027258] zswap: loaded using pool lzo/zbud [ 3.053410] Key type big_key registered [ 3.067174] Key type encrypted registered [ 3.069075] ima: No TPM chip found, activating TPM-bypass! [ 3.071433] ima: Allocated hash algorithm: sha1 [ 3.073320] ima: No architecture policies found [ 3.075391] evm: Initialising EVM extended attributes: [ 3.077295] evm: security.selinux [ 3.078609] evm: security.ima [ 3.079742] evm: security.capability [ 3.081029] evm: HMAC attrs: 0x1 [ 3.083392] rtc_cmos 00:05: setting system clock to 2026-07-16 07:38:03 UTC (1784187483) [ 3.090643] debug: unmapping init [mem 0xffffffff92e03000-0xffffffff92ffffff] [ 3.093977] debug: unmapping init [mem 0xffffffff91b82000-0xffffffff91e58fff] [ 3.106183] Write protecting the kernel read-only data: 28672k [ 3.109694] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.113105] debug: unmapping init [mem 0xffffffff90b14000-0xffffffff90bfffff] [ 3.142496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.150904] systemd[1]: Detected virtualization kvm. [ 3.152985] systemd[1]: Detected architecture x86-64. [ 3.154861] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.184136] systemd[1]: No hostname configured. [ 3.186136] systemd[1]: Set hostname to . [ 3.188281] random: systemd: uninitialized urandom read (16 bytes read) [ 3.191290] systemd[1]: Initializing machine ID from random generator. [ 3.242227] random: ln: uninitialized urandom read (6 bytes read) [ 3.319428] random: systemd: uninitialized urandom read (16 bytes read) [ 3.322590] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.327770] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.331815] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.969053] device-mapper: uevent: version 1.0.3 [ 3.971314] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 4.342119] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.748243] virtio_net virtio0 ens2: renamed from eth0 [ 4.855482] scsi host0: ata_piix [ 4.876551] scsi host1: ata_piix [ 4.878101] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.880540] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.308926] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.592163] random: crng init done [ 9.593527] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.116519] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.291191] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.561121] SELinux: Disabled at runtime. [ 11.623651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.635129] systemd[1]: Detected virtualization kvm. [ 11.637222] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.164438] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.168364] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.174476] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.178803] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.182535] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.190969] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.199131] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.284483] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.635683] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.927233] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.981491] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.012630] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.060045] EDAC sbridge: Ver: 1.1.2 [ 13.991438] Key type dns_resolver registered [ 14.284958] NFS: Registering the id_resolver key type [ 14.286917] Key type id_resolver registered [ 14.288636] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg151-client login: [ 46.976182] libcfs: loading out-of-tree module taints kernel. [ 47.086894] Key type ._llcrypt registered [ 47.100618] Key type .llcrypt registered [ 47.606372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.646210] alg: No test for adler32 (adler32-zlib) [ 49.167982] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 50.231375] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 51.975166] Key type lgssc registered [ 53.943279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.943344] hrtimer: interrupt took 7330702 ns [ 185.786825] Lustre: Mounted lustre-client [ 191.715677] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 205.164337] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.423225] Lustre: lustre-OST0000-osc-ffff8e7f189e1800: disconnect after 22s idle [ 212.664685] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing yml_node [ 220.295823] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 223.142819] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 225.742630] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 227.346459] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jul 16 03:41:46 EDT 2026 [ 243.631717] Lustre: Unmounted lustre-client [ 274.250672] Key type lgssc unregistered [ 274.590195] LNet: 8376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.594874] LNetError: 8376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 275.623891] LNet: Removed LNI 192.168.201.51@tcp [ 276.760222] Key type .llcrypt unregistered [ 276.764887] Key type ._llcrypt unregistered [ 306.821808] Key type ._llcrypt registered [ 306.830446] Key type .llcrypt registered [ 307.122526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.168590] alg: No test for adler32 (adler32-zlib) [ 308.177596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 308.255149] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 308.280563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 313.805449] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 314.345677] LNet: 9262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.362975] LNetError: 9262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.383889] LNet: Removed LNI 192.168.201.51@tcp [ 314.950335] Key type .llcrypt unregistered [ 314.952050] Key type ._llcrypt unregistered [ 325.327149] Key type ._llcrypt registered [ 325.335716] Key type .llcrypt registered [ 326.626758] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:43:25 (1784187805) [ 327.281582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 327.292966] alg: No test for adler32 (adler32-zlib) [ 328.211874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.273964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.311735] LNet: 9673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.346537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.428647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.474058] LNet: 9680:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 337.410459] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:43:36 (1784187816) [ 337.660198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.711471] LNet: 10258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.813307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.006931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 345.547516] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:43:44 (1784187824) [ 345.817533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.855634] LNet: 10845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.883194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.963557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 352.545976] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:43:51 (1784187831) [ 352.677838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.701920] LNet: 11425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.721846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.806924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 359.533736] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:43:58 (1784187838) [ 359.769404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.809426] LNet: 12014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.842127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.035268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 360.551890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 367.803316] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:44:06 (1784187846) [ 367.991637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.009283] LNet: 12604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.052804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.179594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 368.550506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 368.963535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 374.660496] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:44:13 (1784187853) [ 374.773444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.828162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.961370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 381.183687] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:44:20 (1784187860) [ 381.456182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.525637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.589853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 381.645067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 382.118683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 382.165980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 382.575305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 382.623072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 383.067482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 383.101509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 383.586662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 383.656798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 384.152993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 384.222979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 384.691813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 384.725651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 385.226498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 385.298650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 386.001292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 386.064885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 386.304772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 386.328349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 386.581666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 386.606566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 386.882213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 386.915205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.351653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.417482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 394.621429] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:44:33 (1784187873) [ 394.749884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.782432] LNet: 14462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.798450] LNet: 14462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 394.829611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.990809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 395.040325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 403.435807] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:44:41 (1784187881) [ 403.570645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.637131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.799119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 403.834687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 410.592706] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:44:49 (1784187889) [ 410.792727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.894654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.034846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 411.062315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 417.634276] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:44:56 (1784187896) [ 417.751281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.811226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.924527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 417.968470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 425.531328] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:45:04 (1784187904) [ 425.761991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.837328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.955934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 426.001412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 433.013669] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:45:12 (1784187912) [ 433.102460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.158641] LNet: 17395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.178628] LNet: 17395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 433.234787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.319788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 433.358568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 440.649735] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:45:19 (1784187919) [ 440.771912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.861154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.958674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.456158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.414161] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:45:26 (1784187926) [ 447.589817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.679322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.941379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.312466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.285823] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:45:34 (1784187934) [ 455.511603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.564967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.686790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.990461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.268588] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:45:42 (1784187942) [ 463.522191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.567441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.115862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.335547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.409685] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:45:49 (1784187949) [ 470.505436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.543406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.632081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.682883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.836831] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:45:56 (1784187956) [ 478.006366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.087905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.227153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.329630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.290300] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:46:04 (1784187964) [ 485.505331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.557250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.627767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 491.969270] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:46:11 (1784187971) [ 492.075692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.138276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.252975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 498.665372] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:46:17 (1784187977) [ 498.764090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.804445] LNet: 22832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.810639] LNet: 22832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 498.862551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.935621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 499.389331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 507.204422] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:46:25 (1784187985) [ 507.343359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.411358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.583835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 507.626344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 514.184539] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:46:33 (1784187993) [ 514.331107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.397687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.470729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 514.494499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 521.288576] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:46:40 (1784188000) [ 521.419216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.507223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.665811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 521.752705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 527.895574] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:46:47 (1784188007) [ 527.992653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.030875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.148878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 528.184643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 535.007364] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:46:53 (1784188013) [ 535.125668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.193481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.292694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 535.384541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 535.411982] LNetError: 25787:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 535.473831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 535.521825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 541.927733] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:47:01 (1784188021) [ 542.097507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.160430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.245502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5hTtCpFxnP [ 542.308961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5hTtCpFxnP [ 542.383993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.5hTtCpFxnP [ 548.635970] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:47:07 (1784188027) [ 548.845091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.923670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.009191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 549.053455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 549.098544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 555.111892] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 03:47:14 (1784188034) [ 555.249806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.302036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.432325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 555.487071] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 557.755959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 559.515265] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 561.463163] Key type lgssc registered [ 562.806791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 567.987977] LNetError: 27546:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.201.51@tcp99 has no available nets [ 571.683051] Key type lgssc unregistered [ 572.363260] LNetError: 28657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 572.396821] LNet: Removed LNI 192.168.201.51@tcp [ 573.408600] Key type .llcrypt unregistered [ 573.409719] Key type ._llcrypt unregistered [ 582.650125] Key type ._llcrypt registered [ 582.656356] Key type .llcrypt registered [ 583.911742] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 03:47:42 (1784188062) [ 584.325278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 584.337452] alg: No test for adler32 (adler32-zlib) [ 585.282780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.369263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 588.414182] LNet: Message forwarding will be enabled in 2s [ 591.519146] LNet: Message forwarding enabled [ 591.979592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.992682] LNet: Message forwarding will stop in 2s to 6s [ 595.478563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.596436] LNet: Message forwarding disabled [ 599.025537] LNet: Message forwarding will be enabled in 2s [ 602.787466] LNet: Message forwarding enabled [ 607.110592] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 03:48:06 (1784188086) [ 607.505708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.546175] LNet: 30057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.088572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.145328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 615.274186] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 03:48:14 (1784188094) [ 615.399810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.429351] LNet: 30635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.464831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.525794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 615.562948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 615.587754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 615.610456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 615.650252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 615.688276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 615.731595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 615.773289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 615.806454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 615.843073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 615.884885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 615.918640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 615.968299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 616.013083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 616.090474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 616.145343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 616.198701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 616.236429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 616.278218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 616.297634] LNetError: 30681:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 622.150818] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 03:48:21 (1784188101) [ 622.263853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.301503] LNet: 31247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.345775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.448798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.062442] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 03:48:28 (1784188108) [ 630.679991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg151-server [ 632.449848] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 03:48:31 (1784188111) [ 632.555878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.593960] LNet: 32175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.639813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.738687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.769914] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 633.476880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[52-56]@tcp [ 633.490789] LNet: 32316:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 633.550115] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.53@tcp is being used as a gateway but routing feature is not turned on [ 633.567326] LNetError: 32184:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.53@tcp has gone from up to down [ 634.559656] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 634.588360] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 635.577497] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 635.588832] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 637.604319] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.54@tcp is being used as a gateway but routing feature is not turned on [ 637.623214] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 640.341225] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 03:48:39 (1784188119) [ 641.722734] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 641.750498] LNetError: 32184:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 642.106848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg151-server [ 643.526497] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 03:48:42 (1784188122) [ 643.890470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.912447] LNet: 33252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.924965] LNetError: 33252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.945731] LNet: Removed LNI 192.168.201.51@tcp [ 643.966441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.034514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.069729] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 645.147280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[52-57/2]@tcp [ 645.170248] LNet: 33369:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 645.181229] LNet: 33369:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 645.215571] LNetError: 33256:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.54@tcp has gone from up to down [ 645.217964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[52-57/2]@tcp [ 645.233747] LNetError: 33256:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 652.477522] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 03:48:51 (1784188131) [ 652.714569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.748847] LNet: 33962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.781034] LNetError: 33962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.807585] LNet: Removed LNI 192.168.201.51@tcp [ 652.824924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.952783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 653.025415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.063226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.155830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.223909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.266859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.337524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 653.406345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.445473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.532525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.652470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.712935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.801743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 653.901754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.976939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.087854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.215101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.257724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.322350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 654.446737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.491384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.606854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.674988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.716308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.829781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 654.926505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.955789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.048889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.175731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.225993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.295805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 655.340879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.368721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.413877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.898187] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 03:49:01 (1784188141) [ 664.250085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg151-server [ 665.939275] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 03:49:05 (1784188145) [ 667.979159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg151-server [ 669.886333] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 03:49:08 (1784188148) [ 669.983457] LNet: 35355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.996675] LNet: 35355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 670.592983] Key type .llcrypt unregistered [ 670.598207] Key type ._llcrypt unregistered [ 677.996621] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 678.095645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.102702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 678.122810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 681.168544] Key type ._llcrypt registered [ 681.170594] Key type .llcrypt registered [ 681.365526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.374676] alg: No test for adler32 (adler32-zlib) [ 682.409879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.526543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.588376] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 683.576401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 683.620054] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.465294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 686.496491] LNet: Removed LNI 10.1.2.3@tcp [ 686.669708] LNet: 36086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.676923] LNetError: 36086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.692469] LNet: Removed LNI 192.168.201.51@tcp [ 687.486827] Key type .llcrypt unregistered [ 687.492941] Key type ._llcrypt unregistered [ 703.174716] Key type ._llcrypt registered [ 703.176174] Key type .llcrypt registered [ 704.544670] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 03:49:43 (1784188183) [ 705.174536] Key type .llcrypt unregistered [ 705.179256] Key type ._llcrypt unregistered [ 711.990772] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.131567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.142204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.157061] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.708110] Key type ._llcrypt registered [ 714.714337] Key type .llcrypt registered [ 714.955210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.977992] alg: No test for adler32 (adler32-zlib) [ 715.925363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.985070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.016082] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 716.701443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 716.715867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.567661] LNet: 37599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.575887] LNetError: 37599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 718.582114] LNet: Removed LNI 10.1.2.3@tcp [ 718.592025] LNet: Removed LNI 192.168.201.51@tcp [ 719.032572] Key type .llcrypt unregistered [ 719.034877] Key type ._llcrypt unregistered [ 729.706385] Key type ._llcrypt registered [ 729.709194] Key type .llcrypt registered [ 731.090219] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 03:50:09 (1784188209) [ 731.704564] Key type .llcrypt unregistered [ 731.707639] Key type ._llcrypt unregistered [ 738.249881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.559464] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.564228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.583804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.279947] Key type ._llcrypt registered [ 741.286698] Key type .llcrypt registered [ 741.641698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.666914] alg: No test for adler32 (adler32-zlib) [ 742.657880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.767740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 742.824605] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 745.802039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 745.895976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.988832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 749.025726] LNet: Removed LNI 10.1.2.3@tcp [ 749.045973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 749.082936] LNet: Removed LNI 10.1.2.31@tcp [ 749.099535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 749.121739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 750.020878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 750.043722] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 751.191785] LNet: 39523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.198417] LNetError: 39523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 751.203374] LNet: Removed LNI 10.1.2.31@tcp [ 751.217239] LNet: Removed LNI 10.1.2.3@tcp [ 752.049453] Key type .llcrypt unregistered [ 752.064328] Key type ._llcrypt unregistered [ 766.739549] Key type ._llcrypt registered [ 766.743625] Key type .llcrypt registered [ 768.253305] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 03:50:46 (1784188246) [ 768.637253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 768.655913] alg: No test for adler32 (adler32-zlib) [ 769.518272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.560566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.576833] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 770.168166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.186649] LNet: 40477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.191638] LNetError: 40477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.207221] LNet: Removed LNI 192.168.201.51@tcp [ 770.224351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.263621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.51@tcp [ 770.284261] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 770.838323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.51@tcp [ 770.864254] LNet: Removed LNI 192.168.201.51@tcp [ 770.918886] LNet: 40612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.455441] Key type .llcrypt unregistered [ 771.457554] Key type ._llcrypt unregistered [ 783.728995] Key type ._llcrypt registered [ 783.731778] Key type .llcrypt registered [ 785.380776] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 03:51:03 (1784188263) [ 787.551147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg151-server [ 789.526475] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 03:51:08 (1784188268) [ 790.121641] Key type .llcrypt unregistered [ 790.123467] Key type ._llcrypt unregistered [ 799.393949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 799.499134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 799.509215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 799.521919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 802.426491] Key type ._llcrypt registered [ 802.433093] Key type .llcrypt registered [ 802.702610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.715454] alg: No test for adler32 (adler32-zlib) [ 803.790234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.971675] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 805.034421] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 807.858355] LNet: 42452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.871162] LNetError: 42452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.879652] LNet: Removed LNI 192.168.201.51@tcp1 [ 807.901357] LNet: Removed LNI 10.1.2.3@tcp2 [ 808.453514] Key type .llcrypt unregistered [ 808.457044] Key type ._llcrypt unregistered [ 822.440575] Key type ._llcrypt registered [ 822.443383] Key type .llcrypt registered [ 823.619622] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 03:51:42 (1784188302) [ 824.168546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 824.177132] alg: No test for adler32 (adler32-zlib) [ 825.134828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.181805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.199365] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 825.968530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.989903] LNet: 43422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.995412] LNetError: 43422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.009547] LNet: Removed LNI 192.168.201.51@tcp [ 826.019584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.062106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 826.084759] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 831.955114] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 03:51:51 (1784188311) [ 832.121816] LNet: 44009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.135751] LNetError: 44009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.171350] LNet: Removed LNI 192.168.201.51@tcp [ 832.633532] Key type .llcrypt unregistered [ 832.635292] Key type ._llcrypt unregistered [ 838.408847] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 838.699495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 838.706054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 838.709194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 840.524513] Key type ._llcrypt registered [ 840.536307] Key type .llcrypt registered [ 840.944723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.979728] alg: No test for adler32 (adler32-zlib) [ 841.937494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.030445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.078954] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 843.090042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 843.110661] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.972316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.986819] LNet: 44748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.998401] LNetError: 44748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 845.001648] LNet: Removed LNI 10.1.2.3@tcp [ 845.011760] LNet: Removed LNI 192.168.201.51@tcp [ 845.033327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.085452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 845.111798] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 845.114948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.483632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.502919] LNet: 44775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.507664] LNetError: 44775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 845.511754] LNetError: 44775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.516587] LNet: Removed LNI 10.1.2.3@tcp [ 845.522718] LNet: Removed LNI 192.168.201.51@tcp [ 845.538703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.612900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 845.644587] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 845.648205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 846.107319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 846.125995] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 847.053443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.070553] LNet: 44937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.075738] LNetError: 44937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 847.080851] LNetError: 44937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 847.085912] LNet: Removed LNI 10.1.2.3@tcp [ 847.088191] LNet: Removed LNI 192.168.201.51@tcp [ 847.106786] LNet: Removed LNI 192.168.201.51@tcp2 [ 847.152410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.229468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 847.267536] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 847.270961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 847.279741] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 854.091782] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 03:52:13 (1784188333) [ 854.519140] LNet: 45535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.528232] LNetError: 45535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 854.536831] LNetError: 45535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 854.545248] LNet: Removed LNI 10.1.2.3@tcp [ 854.548804] LNet: Removed LNI 192.168.201.51@tcp [ 854.557284] LNet: Removed LNI 192.168.201.51@tcp2 [ 855.266555] Key type .llcrypt unregistered [ 855.271590] Key type ._llcrypt unregistered [ 862.073808] Key type ._llcrypt registered [ 862.078336] Key type .llcrypt registered [ 862.328039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.353797] alg: No test for adler32 (adler32-zlib) [ 863.302314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.354840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.381640] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 864.452251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.468488] LNet: 45929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.473806] LNetError: 45929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.516970] LNet: Removed LNI 192.168.201.51@tcp [ 864.543070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.640802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.664681] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 865.083606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.100557] LNet: 45959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.106764] LNetError: 45959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.124602] LNet: Removed LNI 192.168.201.51@tcp [ 865.143654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.192281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 865.213175] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 865.583990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.613514] LNet: Removed LNI 192.168.201.51@tcp [ 865.628478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.676388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 865.698063] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 866.096950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 866.335056] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 866.341679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 866.349512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 868.662347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.678951] LNet: 46070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.683421] LNet: 46070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 868.688178] LNetError: 46070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.692177] LNetError: 46070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 868.717353] LNet: Removed LNI 192.168.201.51@tcp [ 868.739636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.809809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 868.834798] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 874.881233] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 03:52:33 (1784188353) [ 875.000242] LNet: 46658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.009949] LNetError: 46658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.040693] LNet: Removed LNI 192.168.201.51@tcp [ 875.577903] Key type .llcrypt unregistered [ 875.585781] Key type ._llcrypt unregistered [ 881.280754] Key type ._llcrypt registered [ 881.286372] Key type .llcrypt registered [ 881.579762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.609026] alg: No test for adler32 (adler32-zlib) [ 882.553769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.626338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.647667] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 883.396677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.433165] LNet: 47065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.450932] LNetError: 47065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.473723] LNet: Removed LNI 192.168.201.51@tcp [ 883.523320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.603894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 883.638092] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 889.983526] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 03:52:49 (1784188369) [ 890.165839] LNet: 47654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.176065] LNetError: 47654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.206767] LNet: Removed LNI 192.168.201.51@tcp [ 890.773497] Key type .llcrypt unregistered [ 890.778073] Key type ._llcrypt unregistered [ 896.979501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.235420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 897.246244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 897.257759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 899.945718] Key type ._llcrypt registered [ 899.954226] Key type .llcrypt registered [ 900.169980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.218159] alg: No test for adler32 (adler32-zlib) [ 901.204040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.280479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 901.298701] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 902.249429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.291249] LNet: 48107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.305130] LNetError: 48107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.325268] LNet: Removed LNI 192.168.201.51@tcp [ 902.362928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.490861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 902.542105] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 911.392362] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 03:53:09 (1784188389) [ 911.725924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.781538] LNet: 48708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.798034] LNetError: 48708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.818492] LNet: Removed LNI 192.168.201.51@tcp [ 911.845307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.994697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 918.524760] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 03:53:17 (1784188397) [ 918.626544] LNet: 49282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.389812] Key type .llcrypt unregistered [ 919.395883] Key type ._llcrypt unregistered [ 925.678625] Key type ._llcrypt registered [ 925.682339] Key type .llcrypt registered [ 926.004848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.025806] alg: No test for adler32 (adler32-zlib) [ 927.006404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.093432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.144265] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 927.919021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.937715] LNet: 49688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.944729] LNetError: 49688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.968305] LNet: Removed LNI 192.168.201.51@tcp [ 927.990569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.080424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 928.098971] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 934.489083] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 03:53:33 (1784188413) [ 934.606943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.629244] LNet: 50274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.641282] LNetError: 50274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.668398] LNet: Removed LNI 192.168.201.51@tcp [ 934.692753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.786832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 941.192622] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 03:53:40 (1784188420) [ 943.248930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 945.562746] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 03:53:44 (1784188424) [ 947.708899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 949.127473] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 03:53:48 (1784188428) [ 951.008479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 952.899556] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 03:53:51 (1784188431) [ 954.734274] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 956.641899] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 03:53:55 (1784188435) [ 958.523910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 960.294740] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 03:53:59 (1784188439) [ 960.611815] LNet: 52334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.155797] Key type .llcrypt unregistered [ 961.157435] Key type ._llcrypt unregistered [ 968.567624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 968.925082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 968.931775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 968.950437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 971.660584] Key type ._llcrypt registered [ 971.667470] Key type .llcrypt registered [ 971.997158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.027113] alg: No test for adler32 (adler32-zlib) [ 973.029787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.189439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 973.228770] LNet: Added LNI 192.168.201.51@tcp [8/512/0/180] [ 973.239238] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 973.241022] LNet: 52651:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 973.324110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.350909] LNet: 52670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.370133] LNetError: 52670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 973.379185] LNet: Removed LNI 10.1.2.3@tcp [ 973.389965] LNetError: 52670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.412640] LNet: Removed LNI 192.168.201.51@tcp [ 973.459359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.612694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 973.678253] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 973.784376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.819445] LNet: 52694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.825243] LNetError: 52694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.841476] LNet: Removed LNI 192.168.201.51@tcp [ 973.867201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.953866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 973.981116] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 973.992910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 973.997604] LNet: 52702:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 974.093605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.146545] LNet: 52720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.168693] LNetError: 52720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 974.182971] LNet: Removed LNI 10.1.2.3@tcp [ 974.186777] LNetError: 52720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.229628] LNet: Removed LNI 192.168.201.51@tcp [ 974.276540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.389888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 974.430519] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 974.436614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 974.439861] LNet: 52728:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 974.678311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.706817] LNet: 52754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.715234] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 974.721799] LNet: Removed LNI 10.1.2.3@tcp [ 974.723550] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.753215] LNet: Removed LNI 192.168.201.51@tcp [ 974.783728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.866391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 974.901785] LNet: Added LNI 192.168.201.51@tcp [8/256/0/182] [ 974.915642] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 975.056414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.098333] LNet: 52783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.117928] LNetError: 52783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 975.130152] LNet: Removed LNI 10.1.2.3@tcp [ 975.131794] LNetError: 52783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.188717] LNet: Removed LNI 192.168.201.51@tcp [ 975.255758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.369319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 975.400866] LNet: Added LNI 192.168.201.51@tcp [8/256/0/182] [ 975.406952] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 975.408962] LNet: 52791:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 975.629823] LNet: Removed LNI 10.1.2.3@tcp [ 975.656062] LNet: Removed LNI 192.168.201.51@tcp [ 976.418883] Key type .llcrypt unregistered [ 976.422169] Key type ._llcrypt unregistered [ 992.207718] Key type ._llcrypt registered [ 992.210791] Key type .llcrypt registered [ 993.887247] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 03:54:32 (1784188472) [ 994.527445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.552915] alg: No test for adler32 (adler32-zlib) [ 995.647621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.694743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 995.717175] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 996.654542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.51 [ 996.682665] LNet: 53768:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 996.698533] LNet: Removed LNI 192.168.201.51@tcp [ 1005.003848] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 03:54:43 (1784188483) [ 1005.250857] LNet: 54335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.256876] LNetError: 54335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.300778] LNet: Removed LNI 192.168.201.51@tcp [ 1005.889630] Key type .llcrypt unregistered [ 1005.895049] Key type ._llcrypt unregistered [ 1013.064238] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.350570] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1015.177807] Key type ._llcrypt registered [ 1015.180883] Key type .llcrypt registered [ 1015.395665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.432628] alg: No test for adler32 (adler32-zlib) [ 1017.442862] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.582623] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 03:55:03 (1784188503) [ 1024.823552] LNet: 55245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.833695] LNetError: 55245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.846562] LNet: Removed LNI 10.1.2.3@tcp [ 1025.456610] Key type .llcrypt unregistered [ 1025.459060] Key type ._llcrypt unregistered [ 1032.931847] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.161762] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.169328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.176390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1034.906535] Key type ._llcrypt registered [ 1034.908759] Key type .llcrypt registered [ 1035.133495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.148474] alg: No test for adler32 (adler32-zlib) [ 1036.116548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.257950] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 03:55:23 (1784188523) [ 1046.087466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1047.874052] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 03:55:26 (1784188526) [ 1048.005242] LNet: 56417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.013956] LNetError: 56417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.048820] LNet: Removed LNI 10.1.2.3@tcp [ 1048.809838] Key type .llcrypt unregistered [ 1048.813122] Key type ._llcrypt unregistered [ 1055.484823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1055.710665] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1055.716346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1055.725680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1057.336764] Key type ._llcrypt registered [ 1057.340829] Key type .llcrypt registered [ 1057.590391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1057.610270] alg: No test for adler32 (adler32-zlib) [ 1058.558827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.639621] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.586306] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 03:55:47 (1784188547) [ 1069.046489] LNet: 57645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.050493] LNetError: 57645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.062624] LNet: Removed LNI 10.1.2.3@tcp [ 1069.683951] Key type .llcrypt unregistered [ 1069.685797] Key type ._llcrypt unregistered [ 1076.650738] Key type ._llcrypt registered [ 1076.654941] Key type .llcrypt registered [ 1077.038929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.080575] alg: No test for adler32 (adler32-zlib) [ 1078.045743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1078.075588] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1087.574841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1088.286939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1088.327694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1088.377333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1088.400505] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1088.742921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.781866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.845534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.911118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.959922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.997925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.054373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.130174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.172168] LNet: 58083:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.212536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.272588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.351496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.375764] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1089.377444] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1089.399122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.432071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.476696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.507555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1090.062373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1090.100108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1090.140340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1090.407898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.458539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.500554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.544369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.590434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.635584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.701322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.736513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.756348] LNet: 58192:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.772318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.828225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.884340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.961220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.014570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.096230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.167918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1092.009459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1092.062637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1092.131460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1092.174556] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1092.192674] LNet: Skipped 1 previous similar message [ 1092.545208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.597480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1092.647181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.679365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.714992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.758089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.790856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.859500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.896331] LNet: 58302:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.941302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.988644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.024561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.052861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.081249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.112753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.140818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1093.678236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1093.713428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1093.744897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1093.992862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.017920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.046699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.075483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.104900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.134761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.164820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.201244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.240962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.268891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.297407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.319169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.341569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.363409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1094.392430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.929138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1094.979677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1095.038076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1095.075166] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1095.083728] LNet: Skipped 1 previous similar message [ 1095.333184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.365176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1095.398177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.426651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.460253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.488706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.522820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.553662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.570777] LNet: 58520:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.575070] LNet: 58520:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1095.590713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.624930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.653449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.708265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.746738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.800174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.834647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.461797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1096.509351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1096.558814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1096.771408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.812207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1096.846780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.883770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.930832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.962972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.991100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.019569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.044677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.078068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.111322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.136584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.183826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.241234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.285927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1097.661472] LNet: 58670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.675615] LNetError: 58670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.737436] LNet: Removed LNI 192.168.201.51@tcp [ 1099.213274] Key type .llcrypt unregistered [ 1099.216943] Key type ._llcrypt unregistered [ 1118.232869] Key type ._llcrypt registered [ 1118.235617] Key type .llcrypt registered [ 1119.607510] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 03:56:38 (1784188598) [ 1120.663323] Key type .llcrypt unregistered [ 1120.664602] Key type ._llcrypt unregistered [ 1127.899593] Key type ._llcrypt registered [ 1127.903387] Key type .llcrypt registered [ 1128.229527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.274208] alg: No test for adler32 (adler32-zlib) [ 1129.187714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1129.214158] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1138.087345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1150.713573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.773872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1150.813918] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1152.309294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1152.345714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 1 [ 1152.376116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1152.406858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 1 [ 1152.439317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1152.473039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 1 [ 1152.514153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1152.561922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 1 [ 1152.630547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1152.658579] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1152.666546] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1152.915524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.980957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.022613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.086882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.147428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.201702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.238168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.291313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.308623] LNet: 60139:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.333254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.396364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.453887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.498113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.539071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.610232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.667769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1154.336485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1154.376397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 1 [ 1154.403377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1154.434895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 1 [ 1154.463665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1154.492211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 1 [ 1154.519357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1154.543949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 1 [ 1154.585379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1154.631951] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1154.636118] LNet: Skipped 2 previous similar messages [ 1154.848898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.883254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.914297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.943133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.976672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.017119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.055981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.085719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.108902] LNet: 60261:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.168876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.198918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.252228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.283470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.347542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.392842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.441639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1156.089404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1156.131736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 1 [ 1156.178597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1156.216194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 1 [ 1156.277400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1156.323672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 1 [ 1156.367778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1156.413837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 1 [ 1156.477894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1156.505659] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1156.511634] LNet: Skipped 2 previous similar messages [ 1156.819213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.850316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.890488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.928691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.970217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.009560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.051948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.088156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.114546] LNet: 60383:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.130263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.168347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.217454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.256060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.297330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.336799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.372977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1157.985698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1158.021961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 1 [ 1158.045449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1158.073735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 1 [ 1158.100922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1158.129600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 1 [ 1158.160100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1158.185910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 1 [ 1158.218737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1158.519216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.550442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.605320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.652699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.714866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.776624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.838973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.880470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.918673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.972026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.018944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.069967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.108378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.144406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.178614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1159.787224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1159.823405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 1 [ 1159.853574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1159.884699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 1 [ 1159.916693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1159.954765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 1 [ 1159.987381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1160.035607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 1 [ 1160.066093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1160.083730] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1160.089380] LNet: Skipped 5 previous similar messages [ 1160.320697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1160.346691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1160.372663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1160.401548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1160.433956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1160.466891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1160.492468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.523567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.552447] LNet: 60626:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.556740] LNet: 60626:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1160.574381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.611348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.643268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.679995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.721853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.763386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.795174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.471699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1161.526903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_error -r 1 [ 1161.593332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1161.627684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_error -r 1 [ 1161.688634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1161.734776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_error -r 1 [ 1161.803986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1161.851394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_error -r 1 [ 1161.899644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1162.240974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.296802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.323352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.357909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.382991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.443593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.503343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.559807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.609409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.693371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.764857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.801743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.853583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.907232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.956272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.221373] LNet: 60823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.228412] LNetError: 60823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.236350] LNet: Removed LNI 192.168.201.51@tcp [ 1170.278719] LNet: Removed LNI 192.168.201.51@tcp1 [ 1170.825980] Key type .llcrypt unregistered [ 1170.837058] Key type ._llcrypt unregistered [ 1189.550167] Key type ._llcrypt registered [ 1189.558360] Key type .llcrypt registered [ 1190.896274] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 03:57:49 (1784188669) [ 1191.753521] Key type .llcrypt unregistered [ 1191.756670] Key type ._llcrypt unregistered [ 1197.881800] Key type ._llcrypt registered [ 1197.884706] Key type .llcrypt registered [ 1198.040409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.051554] alg: No test for adler32 (adler32-zlib) [ 1199.036384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.079428] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1207.356884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1208.098497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1208.162123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1208.192352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1208.215914] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1208.483569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.537137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.596932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.688263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1208.743943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1208.782983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1208.804721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1208.829875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1208.854973] LNet: 62072:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1208.882619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1208.940230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.986234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.029284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.058635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.107574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.148517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1209.714538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1209.747351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1209.792128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1209.854513] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1210.091623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.118678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.149363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.176338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.220973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.257051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.285646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.313953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.330630] LNet: 62181:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.351506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.382692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.409732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.441491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.468422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.493677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.529581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1211.131357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1211.181227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1211.216784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1211.237750] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1211.551887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.585861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.619291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.642694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.665424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.693296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.724746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.758983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.781883] LNet: 62290:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.801344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.823895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.849133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.890785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.952961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.016172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.043569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.376680] LNet: 62331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.383979] LNetError: 62331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.407972] LNet: Removed LNI 192.168.201.51@tcp [ 1212.843717] Key type .llcrypt unregistered [ 1212.848991] Key type ._llcrypt unregistered [ 1230.773404] Key type ._llcrypt registered [ 1230.775705] Key type .llcrypt registered [ 1231.901328] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 03:58:31 (1784188711) [ 1232.717195] Key type .llcrypt unregistered [ 1232.721226] Key type ._llcrypt unregistered [ 1239.408573] Key type ._llcrypt registered [ 1239.413331] Key type .llcrypt registered [ 1239.765468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.803860] alg: No test for adler32 (adler32-zlib) [ 1240.752780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.783434] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1250.150068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1260.237476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.303900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1260.354845] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1262.561245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1262.641462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 1 [ 1262.709812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1262.747801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 1 [ 1262.790732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1262.836347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 1 [ 1262.866249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1262.911976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 1 [ 1262.968839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1263.007030] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1263.017969] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1263.413199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.447700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1263.522867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.581317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.610249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.655610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.692935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.726078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.754344] LNet: 63799:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.777911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.818885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.845192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.884687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.930441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.009019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.081439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.177721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.161892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1265.214151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_error -r 1 [ 1265.268481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1265.321783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_error -r 1 [ 1265.359980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1265.395735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_error -r 1 [ 1265.441575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1265.476709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_error -r 1 [ 1265.523410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1265.551765] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1265.556243] LNet: Skipped 2 previous similar messages [ 1265.863810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1265.914880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1265.959926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.998973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.026183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.114326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.160907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.229671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.265729] LNet: 63922:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1266.298253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.350416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1266.407746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.443436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.485413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.505686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1266.561289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1266.617927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.437652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1267.530401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_timeout -r 1 [ 1267.648276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1267.771890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_timeout -r 1 [ 1267.910568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1268.025363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_timeout -r 1 [ 1268.125783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1268.253382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_timeout -r 1 [ 1268.326309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1268.357642] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1268.373586] LNet: Skipped 1 previous similar message [ 1268.700630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.758222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.820600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.851843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.899948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.014448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.095304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.155962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.187269] LNet: 64045:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.211705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.285762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.355575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.425649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.500192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.576072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.682189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1269.770600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.603978] LNet: 64123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.614745] LNetError: 64123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.625580] LNet: Removed LNI 192.168.201.51@tcp [ 1276.643653] LNet: Removed LNI 192.168.201.51@tcp1 [ 1277.297641] Key type .llcrypt unregistered [ 1277.299381] Key type ._llcrypt unregistered [ 1296.933886] Key type ._llcrypt registered [ 1296.945178] Key type .llcrypt registered [ 1298.482713] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 03:59:37 (1784188777) [ 1300.341673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1302.515806] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 03:59:41 (1784188781) [ 1303.302324] Key type .llcrypt unregistered [ 1303.303563] Key type ._llcrypt unregistered [ 1310.401350] Key type ._llcrypt registered [ 1310.405648] Key type .llcrypt registered [ 1310.702324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1310.730121] alg: No test for adler32 (adler32-zlib) [ 1311.714607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1311.760310] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1319.648305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1320.243111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1320.281845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1320.318694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1320.334855] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1320.556024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.598827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.630482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.657420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.680691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.707416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.734774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.760435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1320.773370] LNet: 65667:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1320.789970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.811779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.833663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.863489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.894862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.925773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1321.339983] LNet: 65706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.353832] LNetError: 65706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.386937] LNet: Removed LNI 192.168.201.51@tcp [ 1321.909154] Key type .llcrypt unregistered [ 1321.913162] Key type ._llcrypt unregistered [ 1344.821395] Key type ._llcrypt registered [ 1344.833827] Key type .llcrypt registered [ 1345.148939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1345.167702] alg: No test for adler32 (adler32-zlib) [ 1346.196952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1346.235988] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1356.471592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1366.961313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.991145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.022534] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1368.608858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1368.635229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e network_timeout -r 1 [ 1368.704919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1368.767386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e network_timeout -r 1 [ 1368.835665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1368.920346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e network_timeout -r 1 [ 1368.976882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1369.042300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e network_timeout -r 1 [ 1369.090694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1369.108076] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1369.304911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1369.326404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1369.348610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1369.382484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.429388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.470584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.504779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.528970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.542088] LNet: 66586:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.554481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.576592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.600948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.625942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.665281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.707553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.039526] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1375.593691] LNet: 66660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.600420] LNetError: 66660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.604257] LNet: Removed LNI 192.168.201.51@tcp [ 1375.625437] LNet: Removed LNI 192.168.201.51@tcp1 [ 1376.067763] Key type .llcrypt unregistered [ 1376.071472] Key type ._llcrypt unregistered [ 1399.345028] Key type ._llcrypt registered [ 1399.347152] Key type .llcrypt registered [ 1400.927499] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:01:19 (1784188879) [ 1401.284649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.298879] alg: No test for adler32 (adler32-zlib) [ 1402.362033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.414586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.459418] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1403.046690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1403.066284] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1403.949034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1404.028639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1404.068821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1404.135723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1404.200362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1404.263604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1404.326217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1404.351520] LNetError: 67775:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1404.453320] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1404.462060] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1406.495401] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1406.935621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1410.591384] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1411.748710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1411.831281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1411.949085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.989283] LNet: 67882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.000441] LNetError: 67882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.010542] LNet: Removed LNI 192.168.201.51@tcp [ 1412.046457] LNet: Removed LNI 192.168.201.51@tcp1 [ 1412.118321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.185383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.219280] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1413.133748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.155712] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1414.347422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1414.406088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.532772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1414.594086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1414.676666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1414.747921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1414.798658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.830581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1414.857976] LNetError: 68155:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1415.200296] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1417.562123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1421.348509] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1421.359640] LNet: Skipped 1 previous similar message [ 1425.976135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1426.035909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1426.093316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.212454] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:01:52 (1784188912) [ 1434.403657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.460740] LNet: 68856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.474493] LNetError: 68856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.485956] LNetError: 68856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1434.499526] LNet: Removed LNI 192.168.201.51@tcp [ 1434.544122] LNet: Removed LNI 192.168.201.51@tcp1 [ 1434.611604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.721377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.749790] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1435.493405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1435.549440] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1436.652422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1436.757230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1436.791382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1436.842898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1436.898352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1436.939544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1437.013723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 0 [ 1437.184637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.729374] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.51@tcp [ 1438.759851] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1438.768557] LNet: Skipped 1 previous similar message [ 1452.163200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1452.405363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1452.463860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.515299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 500 [ 1452.626800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1458.581575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1458.700883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.742767] LNet: 69337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.754837] LNetError: 69337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.758045] LNetError: 69337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1458.781300] LNet: Removed LNI 192.168.201.51@tcp [ 1458.793851] LNet: Removed LNI 192.168.201.51@tcp1 [ 1458.852178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.967477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.008255] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1460.105367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1460.148301] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1461.454543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1461.538859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1461.632904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1461.709338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1461.763704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1461.809352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1461.848391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1461.915834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 0 [ 1463.010025] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1463.012020] LNet: Skipped 3 previous similar messages [ 1463.231930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1474.286342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1474.314455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1474.380456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.708419] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:02:42 (1784188962) [ 1484.028607] LNet: 70383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.035188] LNetError: 70383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.042256] LNetError: 70383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1484.056585] LNet: Removed LNI 192.168.201.51@tcp [ 1484.085959] LNet: Removed LNI 192.168.201.51@tcp1 [ 1485.178205] Key type .llcrypt unregistered [ 1485.186157] Key type ._llcrypt unregistered [ 1493.675992] Key type ._llcrypt registered [ 1493.679173] Key type .llcrypt registered [ 1493.963484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1493.978746] alg: No test for adler32 (adler32-zlib) [ 1494.950531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1494.970379] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1505.631094] Lustre: DEBUG MARKER: Initial discovery [ 1505.689182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1508.745475] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1508.826785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1508.918039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1508.961914] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1509.002777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1509.059492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.151@tcp -r 1 -m GET -l 3 [ 1509.127981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.151@tcp -r 1 -m GET -e local_error [ 1509.163576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.151@tcp -r 1 -m PUT -l 6 [ 1509.348041] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1510.672312] Lustre: DEBUG MARKER: Force oleg151-client.virtnet to discover 192.168.201.151@tcp (in background) [ 1510.753435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1512.344269] Lustre: DEBUG MARKER: Force oleg151-server to discover 192.168.201.51@tcp [ 1513.833433] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1515.241710] Lustre: DEBUG MARKER: Wait for 71141 [ 1516.685965] Lustre: DEBUG MARKER: Finished wait on 71141 [ 1516.913696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1516.951768] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1516.965761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1517.313810] LNet: 71597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.350332] LNetError: 71597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.379225] LNet: Removed LNI 192.168.201.51@tcp [ 1517.913229] Key type .llcrypt unregistered [ 1517.916104] Key type ._llcrypt unregistered [ 1539.060655] Key type ._llcrypt registered [ 1539.064202] Key type .llcrypt registered [ 1540.236983] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:03:39 (1784189019) [ 1540.971078] Key type .llcrypt unregistered [ 1540.973747] Key type ._llcrypt unregistered [ 1548.813514] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1549.010687] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1549.026528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1549.043633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1551.745404] Key type ._llcrypt registered [ 1551.747634] Key type .llcrypt registered [ 1551.970553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1552.017883] alg: No test for adler32 (adler32-zlib) [ 1553.018767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.123624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.157971] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1553.815311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1553.834736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1563.822655] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:04:02 (1784189042) [ 1563.954475] LNet: 73671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.970808] LNetError: 73671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1563.977315] LNet: Removed LNI 10.1.2.3@tcp [ 1563.996757] LNet: Removed LNI 192.168.201.51@tcp [ 1564.620049] Key type .llcrypt unregistered [ 1564.622636] Key type ._llcrypt unregistered [ 1570.535515] Key type ._llcrypt registered [ 1570.539175] Key type .llcrypt registered [ 1570.737760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.768989] alg: No test for adler32 (adler32-zlib) [ 1571.776964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.843217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.878707] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1572.507108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1572.525712] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1573.483898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.51@tcp [ 1574.563080] LNetError: Unexpected error -22 connecting to 192.168.201.51@tcp at host 192.168.201.51:988 [ 1581.071972] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:04:19 (1784189059) [ 1581.220466] LNet: 74803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.225695] LNetError: 74803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1581.233211] LNet: Removed LNI 10.1.2.3@tcp [ 1581.248245] LNet: Removed LNI 192.168.201.51@tcp [ 1581.970789] Key type .llcrypt unregistered [ 1581.973685] Key type ._llcrypt unregistered [ 1589.249235] Key type ._llcrypt registered [ 1589.254958] Key type .llcrypt registered [ 1589.562914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.609190] alg: No test for adler32 (adler32-zlib) [ 1590.600864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.674209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1590.714746] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1591.493645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1591.529538] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 1592.390626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.51@tcp1 --nid 192.168.201.51@tcp2 [ 1597.428574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp1 192.168.201.51@tcp2 [ 1597.479595] LNetError: 75645:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.51@tcp1 to 192.168.201.51@tcp2 - Route table may be misconfigured [ 1597.509708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp2 192.168.201.51@tcp1 [ 1604.344690] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:04:43 (1784189083) [ 1604.476985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.536787] LNet: 76211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.566863] LNetError: 76211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.579259] LNet: Removed LNI 192.168.201.51@tcp1 [ 1605.606987] LNet: Removed LNI 192.168.201.51@tcp2 [ 1605.660859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.756087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.818346] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1606.626768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1606.649071] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1607.664395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1607.720966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.51@tcp [ 1607.808264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp1 192.168.201.51@tcp1 [ 1607.873500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp -d 192.168.201.51@tcp -e network_timeout [ 1607.967706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -e network_timeout [ 1608.033632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -e network_timeout [ 1608.088653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -e network_timeout [ 1608.152627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1608.199365] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1608.233976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1608.335499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1608.412496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1608.868326] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1615.401485] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:04:54 (1784189094) [ 1615.667428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.686718] LNet: 77080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.697345] LNetError: 77080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.703988] LNetError: 77080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1615.709412] LNet: Removed LNI 192.168.201.51@tcp [ 1616.745436] LNet: Removed LNI 192.168.201.51@tcp1 [ 1616.774970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.825867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1617.186368] LNet: 77117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.765478] Key type .llcrypt unregistered [ 1617.767884] Key type ._llcrypt unregistered [ 1631.421561] Key type ._llcrypt registered [ 1631.425706] Key type .llcrypt registered [ 1632.825826] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:05:11 (1784189111) [ 1633.764678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.832270] alg: No test for adler32 (adler32-zlib) [ 1634.755620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.220196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1639.219271] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:05:17 (1784189117) [ 1639.463959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.486397] LNet: 78216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.513728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.578322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.596153] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1640.339826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1640.360553] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1641.134128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1641.171645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp1 [ 1641.202711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp1 [ 1648.387863] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:05:27 (1784189127) [ 1650.314798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg151-server [ 1651.781779] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:05:31 (1784189131) [ 1653.875797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg151-server [ 1655.521835] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:05:34 (1784189134) [ 1657.386945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg151-server [ 1659.332340] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:05:38 (1784189138) [ 1661.771437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg151-server [ 1663.550861] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:05:42 (1784189142) [ 1666.165932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg151-server [ 1667.974802] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:05:47 (1784189147) [ 1669.568837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg151-server [ 1671.449623] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:05:50 (1784189150) [ 1673.369955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg151-server [ 1675.309966] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:05:54 (1784189154) [ 1677.015453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg151-server [ 1678.991265] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:05:57 (1784189157) [ 1681.315807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg151-server [ 1682.927584] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:06:02 (1784189162) [ 1683.127578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.144345] LNet: 82093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.150225] LNetError: 82093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.191358] LNet: Removed LNI 192.168.201.51@tcp [ 1684.196632] LNetError: 82093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.221843] LNet: Removed LNI 192.168.201.51@tcp1 [ 1684.291391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.410279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.437648] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1685.230613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1685.326602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1685.498416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.530422] LNet: 82240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.537212] LNetError: 82240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.570719] LNet: Removed LNI 192.168.201.51@tcp [ 1686.592973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.629667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1686.652395] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1687.351656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1687.499808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.550736] LNet: Removed LNI 192.168.201.51@tcp [ 1688.581677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.646792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.674761] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1689.550643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1689.641172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1689.940780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.984542] LNet: 82533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.991539] LNet: 82533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1690.005490] LNetError: 82533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.012232] LNetError: 82533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1690.036813] LNet: Removed LNI 192.168.201.51@tcp [ 1690.126748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.224911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1690.269317] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1691.315803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1691.648545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.711576] LNet: Removed LNI 192.168.201.51@tcp [ 1692.746114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.858957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1692.883913] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1693.699334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1693.769453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1694.046072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.073398] LNet: 82825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.079512] LNet: 82825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1694.089770] LNetError: 82825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.097992] LNetError: 82825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1694.119196] LNet: Removed LNI 192.168.201.51@tcp [ 1694.136290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.192994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1694.216090] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1694.912413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1695.091204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.166044] LNet: Removed LNI 192.168.201.51@tcp [ 1696.200466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.301898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.342060] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1697.241084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1697.313251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1697.603965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.661892] LNet: Removed LNI 192.168.201.51@tcp [ 1698.705523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.775370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1698.798794] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1699.616687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1699.786885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.845927] LNet: Removed LNI 192.168.201.51@tcp [ 1700.891958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.017791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1701.050727] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1701.990499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1702.065791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1702.356765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.403693] LNet: 83411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.416914] LNet: 83411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1702.432568] LNetError: 83411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.449740] LNetError: 83411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1702.487448] LNet: Removed LNI 192.168.201.51@tcp [ 1702.517910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.621746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1702.676793] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1703.917001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1704.246605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.323364] LNet: Removed LNI 192.168.201.51@tcp [ 1705.365663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.531735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.599325] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1706.580472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1706.732825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1707.316270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.414396] LNet: Removed LNI 192.168.201.51@tcp [ 1708.481633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.575535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1708.606724] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1709.518757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1709.826651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.889349] LNet: Removed LNI 192.168.201.51@tcp [ 1710.907944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.027121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.063573] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1711.838944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1711.908550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.937813] LNet: Removed LNI 192.168.201.51@tcp [ 1711.971761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.043677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.107402] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1712.906542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1712.962943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.985973] LNet: Removed LNI 192.168.201.51@tcp [ 1713.007895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.073509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1713.174894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.217456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.263726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1713.359847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.407171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.462693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1713.573862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.629828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.697834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1713.737523] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1714.430853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.477225] LNet: Removed LNI 192.168.201.51@tcp [ 1714.512698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.567412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1722.523905] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:06:41 (1784189201) [ 1722.630447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.660128] LNet: 84863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.666392] LNet: 84863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1722.687145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.728963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1722.742588] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1724.152960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.169296] LNetError: 85009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.174055] LNetError: 85009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1724.200158] LNet: Removed LNI 192.168.201.51@tcp231 [ 1724.226129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.294624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1724.321667] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1724.330577] LNet: 85017:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1724.844477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1724.900118] LNet: Removed LNI 192.168.201.51@tcp231 [ 1724.926057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1724.976663] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1726.271779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.319389] LNet: Removed LNI 192.168.201.51@tcp231 [ 1726.336845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.418769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1726.487340] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/0] [ 1726.496706] LNet: 85169:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1727.132589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1727.193142] LNet: Removed LNI 192.168.201.51@tcp231 [ 1727.212106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1727.275503] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/0] [ 1728.471981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.519709] LNet: Removed LNI 192.168.201.51@tcp231 [ 1728.555948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.648744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1728.684511] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/60] [ 1728.700700] LNet: 85321:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1729.348353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1729.433885] LNet: Removed LNI 192.168.201.51@tcp231 [ 1729.482573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1729.546149] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/60] [ 1730.924981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.944933] LNet: Removed LNI 192.168.201.51@tcp231 [ 1730.958730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.022370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1731.043065] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1731.049903] LNet: 85472:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.460883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1731.503746] LNet: Removed LNI 192.168.201.51@tcp231 [ 1731.527764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1731.565712] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1742.268357] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:07:00 (1784189220) [ 1742.404685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.446851] LNet: Removed LNI 192.168.201.51@tcp231 [ 1742.473239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.541279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1742.570867] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1743.417584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1743.456122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1743.487233] LNet: Removed LNI 192.168.201.51@tcp232 [ 1743.515475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1743.559169] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1744.706650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1744.789917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1744.852538] LNet: Removed LNI 192.168.201.51@tcp232 [ 1744.892652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1744.937559] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1746.140541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1746.210176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1746.246619] LNet: Removed LNI 192.168.201.51@tcp232 [ 1746.264626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1746.308582] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1747.099978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1747.147875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1747.186727] LNet: Removed LNI 192.168.201.51@tcp232 [ 1747.202294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1754.419450] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:07:13 (1784189233) [ 1755.073508] LNet: 87311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.083379] LNet: 87311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1755.813512] Key type .llcrypt unregistered [ 1755.820442] Key type ._llcrypt unregistered [ 1762.365742] Key type ._llcrypt registered [ 1762.369897] Key type .llcrypt registered [ 1762.608395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1762.626153] alg: No test for adler32 (adler32-zlib) [ 1763.649971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1763.690387] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1772.660426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1784.235641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.286748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1784.332893] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1785.834141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 2 [ 1785.859635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 2 [ 1785.892377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 2 [ 1785.948617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 2 [ 1786.020727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_interrupt -r 2 [ 1786.086615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 2 [ 1786.152488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_interrupt -r 2 [ 1786.185321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 2 [ 1786.219626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1786.237305] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1786.247133] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1786.746090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1786.804134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1786.866511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1786.954255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1787.022630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1787.099765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1787.160394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1787.210561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1787.231513] LNet: 87973:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1787.254974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1787.297242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1787.333568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1787.404260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1787.460184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1787.501899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1788.612688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 2 [ 1788.684735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 2 [ 1788.739615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 2 [ 1788.843494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 2 [ 1788.923323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_dropped -r 2 [ 1788.984590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 2 [ 1789.051169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_dropped -r 2 [ 1789.154498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 2 [ 1789.221970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1789.268529] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1789.690906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.748761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.792931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.851963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.920287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.995247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.051574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.096168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.138138] LNet: 88094:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.159032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.209904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.273698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.326307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.383215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.423710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.965665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 2 [ 1791.002838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 2 [ 1791.042676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 2 [ 1791.121994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 2 [ 1791.205575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_aborted -r 2 [ 1791.243619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 2 [ 1791.291074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_aborted -r 2 [ 1791.326374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 2 [ 1791.377485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1791.405321] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1791.687614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1791.731125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1791.760399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1791.790453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1791.828616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.863602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.914546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.969377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.001958] LNet: 88214:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1792.027813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.057463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.077456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.094787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.111797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.133250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1792.582286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 2 [ 1792.602589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 2 [ 1792.627455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 2 [ 1792.659622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 2 [ 1792.715781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_no_route -r 2 [ 1792.773628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 2 [ 1792.840404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_no_route -r 2 [ 1792.897288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 2 [ 1792.941478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1793.268395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1793.299624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1793.328073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1793.361443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1793.403365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1793.436333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1793.481275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1793.505423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1793.541498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1793.568869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1793.588898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1793.604334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1793.633441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1793.663819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1794.171531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 2 [ 1794.200298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 2 [ 1794.232515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 2 [ 1794.276191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 2 [ 1794.308360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_timeout -r 2 [ 1794.347813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 2 [ 1794.381431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_timeout -r 2 [ 1794.425181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 2 [ 1794.465584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1794.483556] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1794.489898] LNet: Skipped 1 previous similar message [ 1794.831767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1794.873851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1794.913968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1794.967804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1795.022881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1795.089462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1795.165986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1795.197751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1795.223751] LNet: 88455:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1795.229179] LNet: 88455:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1795.262517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1795.316403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1795.355866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1795.404745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1795.461786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1795.496021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1796.025861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 2 [ 1796.057442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 2 [ 1796.096608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 2 [ 1796.136446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 2 [ 1796.172322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_dropped -r 2 [ 1796.207517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 2 [ 1796.285252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_dropped -r 2 [ 1796.338095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 2 [ 1796.408649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1796.443722] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1796.810443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1796.846451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1796.873878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1796.918605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1796.943535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1796.989587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1797.042434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1797.088776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1797.153804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1797.211714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1797.249178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1797.305170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1797.344187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1797.388517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1803.585662] LNet: 88649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.602311] LNetError: 88649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.614041] LNet: Removed LNI 192.168.201.51@tcp [ 1803.641021] LNet: Removed LNI 192.168.201.51@tcp1 [ 1804.061744] Key type .llcrypt unregistered [ 1804.063942] Key type ._llcrypt unregistered [ 1824.223478] Key type ._llcrypt registered [ 1824.225614] Key type .llcrypt registered [ 1825.692648] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:08:24 (1784189304) [ 1826.233607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.262302] alg: No test for adler32 (adler32-zlib) [ 1827.204151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.300619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.335082] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1828.468692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1828.532425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1828.560833] LNetError: 89628:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1828.674666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.51@tcp [ 1828.799717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1835.601786] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:08:34 (1784189314) [ 1835.786777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.833282] LNet: 90218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.840630] LNetError: 90218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.854276] LNet: Removed LNI 192.168.201.51@tcp [ 1835.889434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.003967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.040120] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1837.090747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1837.117581] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1838.030672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.51@tcp [ 1838.049927] LNetError: 90484:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1839.008366] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1839.016677] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1839.321158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.51@tcp [ 1848.440866] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:08:47 (1784189327) [ 1850.464208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg151-server [ 1852.952602] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:08:51 (1784189331) [ 1855.250376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg151-server [ 1856.824532] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:08:55 (1784189335) [ 1858.772908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg151-server [ 1860.956663] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:08:59 (1784189339) [ 1861.132267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.152768] LNet: 92113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.168335] LNetError: 92113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.178799] LNet: Removed LNI 192.168.201.51@tcp [ 1861.195044] LNet: Removed LNI 192.168.201.51@tcp1 [ 1861.231740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.286665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.304338] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1862.100464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1862.176188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1862.237358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1862.336108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1862.369941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1862.406900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1869.846496] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:09:08 (1784189348) [ 1870.231756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.272247] LNet: 92840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.287986] LNetError: 92840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.299271] LNetError: 92840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1870.325133] LNet: Removed LNI 192.168.201.51@tcp [ 1870.357442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.438773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.456584] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1878.014624] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:09:16 (1784189356) [ 1878.145116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.198217] LNet: 93537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.207027] LNetError: 93537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.231515] LNet: Removed LNI 192.168.201.51@tcp [ 1878.270977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.363865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1878.387832] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1887.074990] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:09:25 (1784189365) [ 1889.024698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1891.652524] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:09:30 (1784189370) [ 1892.460605] LNet: 94580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.474219] LNetError: 94580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1892.498084] LNet: Removed LNI 192.168.201.51@tcp [ 1893.232722] Key type .llcrypt unregistered [ 1893.237321] Key type ._llcrypt unregistered [ 1899.932164] Key type ._llcrypt registered [ 1899.934572] Key type .llcrypt registered [ 1900.106730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1900.124212] alg: No test for adler32 (adler32-zlib) [ 1901.111520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1901.157677] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1909.695717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1915.745087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.151@tcp [ 1916.145727] LNet: 94997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.161522] LNetError: 94997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.187793] LNet: Removed LNI 192.168.201.51@tcp [ 1916.779114] Key type .llcrypt unregistered [ 1916.785581] Key type ._llcrypt unregistered [ 1928.772390] Key type ._llcrypt registered [ 1928.775366] Key type .llcrypt registered [ 1930.341547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1932.144276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1934.202862] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:10:13 (1784189413) [ 1936.018961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1938.516471] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:10:16 (1784189416) [ 1940.813638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg151-server [ 1942.343738] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:10:21 (1784189421) [ 1943.948170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg151-server [ 1945.908365] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:10:24 (1784189424) [ 1948.084997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.154865] alg: No test for adler32 (adler32-zlib) [ 1949.136738] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1949.405747] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1951.167178] Key type lgssc registered [ 1952.466127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.790903] Key type lgssc unregistered [ 1974.054799] LNet: 98172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.064082] LNetError: 98172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.107604] LNet: Removed LNI 192.168.201.51@tcp [ 1974.802439] Key type .llcrypt unregistered [ 1974.804079] Key type ._llcrypt unregistered [ 1991.032183] Key type ._llcrypt registered [ 1991.033834] Key type .llcrypt registered [ 1992.571845] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:11:11 (1784189471) [ 1993.201957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.233291] alg: No test for adler32 (adler32-zlib) [ 1994.238910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.364800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.444175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.559964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.609991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2001.684476] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:11:20 (1784189480) [ 2001.810556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.861139] LNet: 99581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.885418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.985785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.042744] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2002.987619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.006934] LNet: 99732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.018399] LNetError: 99732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.034939] LNet: Removed LNI 192.168.201.51@tcp [ 2003.049928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.087465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2003.116919] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2003.550758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.566609] LNetError: 99760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.583328] LNet: Removed LNI 192.168.201.51@tcp [ 2003.600327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.661646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2003.682816] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2010.950340] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:11:29 (1784189489) [ 2011.120246] LNet: 100342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.131669] LNet: 100342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2011.145956] LNetError: 100342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.208515] LNet: Removed LNI 192.168.201.51@tcp [ 2012.011241] Key type .llcrypt unregistered [ 2012.018803] Key type ._llcrypt unregistered [ 2018.602472] Key type ._llcrypt registered [ 2018.606641] Key type .llcrypt registered [ 2018.807564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2018.821200] alg: No test for adler32 (adler32-zlib) [ 2019.801510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.951736] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2019.957857] LNet: *** cfs_fail_loc=e003, val=100*** [ 2020.903180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2020.925520] LNet: *** cfs_fail_loc=e003, val=100*** [ 2020.928757] LNet: Skipped 2 previous similar messages [ 2020.935718] Ping reply received with mapping: [ 2020.938465] found metadata reply NI[192.168.201.51@tcp] [ 2020.941719] 33 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2020.944816] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2020.948133] 5a 5a 5a 5a 5a 5a 5a 5a [ 2021.148883] LNet: 100748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.170252] LNetError: 100748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.195796] LNet: Removed LNI 192.168.201.51@tcp [ 2022.175503] Key type .llcrypt unregistered [ 2022.182167] Key type ._llcrypt unregistered [ 2036.285773] Key type ._llcrypt registered [ 2036.296814] Key type .llcrypt registered [ 2038.022356] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:11:56 (1784189516) [ 2038.588363] Key type .llcrypt unregistered [ 2038.595248] Key type ._llcrypt unregistered [ 2046.020352] Key type ._llcrypt registered [ 2046.022329] Key type .llcrypt registered [ 2046.165326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.178345] alg: No test for adler32 (adler32-zlib) [ 2047.331812] LNetError: 101795:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2047.799765] Key type .llcrypt unregistered [ 2047.803562] Key type ._llcrypt unregistered [ 2053.873174] Key type ._llcrypt registered [ 2053.876139] Key type .llcrypt registered [ 2055.421896] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:12:14 (1784189534) [ 2056.356434] Key type .llcrypt unregistered [ 2056.361751] Key type ._llcrypt unregistered [ 2063.600539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2063.606353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2066.301312] Key type ._llcrypt registered [ 2066.304782] Key type .llcrypt registered [ 2066.497314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2066.519368] alg: No test for adler32 (adler32-zlib) [ 2067.471192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.530867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2078.314437] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:12:37 (1784189557) [ 2078.591443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.629331] LNet: 103577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.639768] LNetError: 103577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2078.681286] LNet: Removed LNI 10.1.2.3@tcp [ 2078.741279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.401655] Key type .llcrypt unregistered [ 2079.406522] Key type ._llcrypt unregistered [ 2087.399890] Key type ._llcrypt registered [ 2087.404245] Key type .llcrypt registered [ 2087.674604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2087.717230] alg: No test for adler32 (adler32-zlib) [ 2088.672654] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2088.732906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2088.884892] LNet: 103866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.900570] LNetError: 103866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.959215] LNet: Removed LNI 192.168.201.51@tcp [ 2090.794533] Key type .llcrypt unregistered [ 2090.803396] Key type ._llcrypt unregistered [ 2104.585309] Key type ._llcrypt registered [ 2104.590061] Key type .llcrypt registered [ 2106.032747] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:13:04 (1784189584) [ 2106.607644] Key type .llcrypt unregistered [ 2106.613097] Key type ._llcrypt unregistered [ 2114.628793] Key type ._llcrypt registered [ 2114.648826] Key type .llcrypt registered [ 2114.993227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2115.010616] alg: No test for adler32 (adler32-zlib) [ 2123.067250] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:13:22 (1784189602) [ 2123.228172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.293588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.394115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.425966] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2124.182690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.209138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2124.253696] LNet: Removed LNI 192.168.201.51@tcp [ 2124.620505] LNet: 105612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.226586] Key type .llcrypt unregistered [ 2125.230308] Key type ._llcrypt unregistered [ 2138.785864] Key type ._llcrypt registered [ 2138.789091] Key type .llcrypt registered [ 2140.630381] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:13:38 (1784189618) [ 2141.326737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.358790] alg: No test for adler32 (adler32-zlib) [ 2142.332250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.418170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.467083] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2143.327506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2143.382926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.51@tcp [ 2143.405226] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2143.413148] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2143.422490] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2143, 0, 0 [ 2143.435155] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2143.439390] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2143.447216] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2143.454138] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2143, 0, 0 [ 2143.460771] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2143.465653] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2143.470247] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2143.473455] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2143, 0, 2193 [ 2143.480910] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2143.485311] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2143.489501] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2143.494347] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2143, 0, 2193 [ 2143.506771] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2150.428085] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:13:49 (1784189629) [ 2150.950822] LNet: 107152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.956051] LNetError: 107152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.977272] LNet: Removed LNI 192.168.201.51@tcp [ 2152.893712] Key type .llcrypt unregistered [ 2152.897062] Key type ._llcrypt unregistered [ 2160.699712] Key type ._llcrypt registered [ 2160.701958] Key type .llcrypt registered [ 2161.041706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2161.069448] alg: No test for adler32 (adler32-zlib) [ 2162.041428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2162.092813] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2171.036424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2181.786656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.846686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2181.893766] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 2183.096406] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2183.400320] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2183.411104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2183.419629] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2185.897905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2185.924669] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2198.245651] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:14:36 (1784189676) [ 2198.380470] LNet: 108658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.390214] LNetError: 108658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.398493] LNet: Removed LNI 192.168.201.51@tcp [ 2199.455627] LNet: Removed LNI 192.168.201.51@tcp1 [ 2199.457546] LNetError: 108658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2199.473295] LNetError: 108658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2199.503806] LNet: Removed LNI 10.1.2.3@tcp99 [ 2200.652747] Key type .llcrypt unregistered [ 2200.657400] Key type ._llcrypt unregistered [ 2208.704943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2208.707909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2211.360423] Key type ._llcrypt registered [ 2211.368514] Key type .llcrypt registered [ 2211.653858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2211.673530] alg: No test for adler32 (adler32-zlib) [ 2212.682572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.734239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.775342] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2213.594863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2213.617710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2215.729543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2215.774493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2215.804511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2215.826594] LNetError: 109409:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2215.869568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2215.929399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2216.530248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2216.640588] LNet: 109429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.648264] LNetError: 109429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2216.654837] LNet: Removed LNI 10.1.2.3@tcp [ 2217.721329] LNet: Removed LNI 192.168.201.51@tcp [ 2218.632517] Key type .llcrypt unregistered [ 2218.637593] Key type ._llcrypt unregistered [ 2230.882663] Key type ._llcrypt registered [ 2230.884609] Key type .llcrypt registered [ 2232.166605] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:15:11 (1784189711) [ 2232.486780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.509187] alg: No test for adler32 (adler32-zlib) [ 2233.466900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.521484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.552242] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2234.287583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2234.407495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg151-client@tcp [ 2242.578464] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:15:21 (1784189721) [ 2242.742322] LNet: 110960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.785505] LNetError: 110960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.842683] LNet: Removed LNI 192.168.201.51@tcp [ 2243.639618] Key type .llcrypt unregistered [ 2243.642323] Key type ._llcrypt unregistered [ 2252.575455] Key type ._llcrypt registered [ 2252.581793] Key type .llcrypt registered [ 2252.843723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.862966] alg: No test for adler32 (adler32-zlib) [ 2253.853617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2253.883235] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2253.939853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2262.592051] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:15:41 (1784189741) [ 2264.548632] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2267.107524] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:15:45 (1784189745) [ 2268.664648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2271.334661] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:15:49 (1784189749) [ 2271.482459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.512238] LNet: 112380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.517561] LNetError: 112380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.537856] LNet: Removed LNI 192.168.201.51@tcp [ 2271.580674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.641291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.673660] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2272.884563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2273.005176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2281.166479] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:15:59 (1784189759) [ 2281.495576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.528764] LNet: 113095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.538932] LNetError: 113095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.565606] LNet: Removed LNI 192.168.201.51@tcp [ 2281.582713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.648374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2281.688887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2282.792344] Key type .llcrypt unregistered [ 2282.803112] Key type ._llcrypt unregistered [ 2298.451854] Key type ._llcrypt registered [ 2298.454242] Key type .llcrypt registered [ 2299.912710] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:16:18 (1784189778) [ 2300.371659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.382634] alg: No test for adler32 (adler32-zlib) [ 2301.379334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.496969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.538412] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2302.958832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2303.088086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2311.548292] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:16:30 (1784189790) [ 2311.642062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.673722] LNet: 114647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.684787] LNetError: 114647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.724239] LNet: Removed LNI 192.168.201.51@tcp [ 2311.760774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.798927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2311.825472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2320.787025] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:16:39 (1784189799) [ 2320.954313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.993595] LNet: 115228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.057588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.165029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2321.688353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2321.729818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2321.823638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2322.353848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2322.397110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2322.548880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.589503] LNet: 115268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.626712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.737749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2323.264424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2323.331195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2323.415777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2323.931771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2323.974632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2332.509566] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:16:51 (1784189811) [ 2333.188650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.262095] LNet: 115869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.329954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.420339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.448546] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2334.066658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2334.084176] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2334.850565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2334.894894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2335.089753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.106641] LNetError: 116148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.113329] LNet: Removed LNI 192.168.201.51@tcp [ 2335.121814] LNet: Removed LNI 192.168.201.51@tcp2 [ 2335.137719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.254781] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2335.272436] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2335.277352] LNet: 116154:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.891791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2336.165956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.182956] LNetError: 116191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.188382] LNetError: 116191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2337.248079] LNet: Removed LNI 192.168.201.51@tcp [ 2337.265889] LNet: Removed LNI 192.168.201.51@tcp2 [ 2337.342439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.431678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.466970] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2338.292069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2338.330558] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2339.347987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2339.386520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2339.433371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2339.626887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.642668] LNet: 116476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.647234] LNet: 116476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2339.653329] LNetError: 116476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.657631] LNetError: 116476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2339.663787] LNet: Removed LNI 192.168.201.51@tcp [ 2340.713673] LNet: Removed LNI 192.168.201.51@tcp2 [ 2340.787714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.911753] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2340.924023] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2340.944935] LNet: 116482:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.440961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2341.732252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.777911] LNet: Removed LNI 192.168.201.51@tcp [ 2342.824589] LNet: Removed LNI 192.168.201.51@tcp2 [ 2342.874721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.992844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.027824] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2344.279394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2344.333833] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2345.643551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2345.769695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2346.261663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.326651] LNetError: 116799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.335629] LNetError: 116799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2346.345327] LNet: Removed LNI 192.168.201.51@tcp [ 2346.368664] LNet: Removed LNI 192.168.201.51@tcp2 [ 2346.433711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.560104] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2346.569102] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2346.576745] LNet: 116805:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.247908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2347.518238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.576586] LNet: Removed LNI 192.168.201.51@tcp [ 2348.594281] LNet: Removed LNI 192.168.201.51@tcp2 [ 2348.637894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.731617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2348.767353] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2349.584411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2349.607820] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2350.681710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2350.743382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2350.832292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2351.119772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.140981] LNet: 117126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.146514] LNet: 117126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2352.164078] LNet: Removed LNI 192.168.201.51@tcp [ 2353.190482] LNet: Removed LNI 192.168.201.51@tcp2 [ 2353.248711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.337988] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2353.349216] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2353.355029] LNet: 117133:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.968436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2354.177564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.204664] LNet: Removed LNI 192.168.201.51@tcp [ 2355.238975] LNet: Removed LNI 192.168.201.51@tcp2 [ 2355.302740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.418425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.459819] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2356.288824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2356.307720] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2357.462669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2357.530810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2357.651797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2358.054755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.088474] LNetError: 117467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.095743] LNetError: 117467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2358.102144] LNet: Removed LNI 192.168.201.51@tcp [ 2358.110469] LNet: Removed LNI 192.168.201.51@tcp2 [ 2358.129933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.206213] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2358.216989] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2358.222230] LNet: 117473:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.668524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2359.087971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.159172] LNet: Removed LNI 192.168.201.51@tcp [ 2360.168955] LNet: Removed LNI 192.168.201.51@tcp2 [ 2360.192233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.240180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.276546] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2361.512944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2361.564805] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2362.818922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2362.849695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2362.876884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2362.921910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2363.272424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.319175] LNet: Removed LNI 192.168.201.51@tcp [ 2365.351287] LNet: Removed LNI 192.168.201.51@tcp2 [ 2365.429759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.534929] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2365.544368] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2366.342438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2375.212726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2376.931704] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:17:35 (1784189855) [ 2377.085760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.121704] LNet: 118591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.132241] LNet: 118591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2377.146172] LNetError: 118591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.152893] LNetError: 118591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2377.157731] LNet: Removed LNI 192.168.201.51@tcp [ 2378.218940] LNet: Removed LNI 192.168.201.51@tcp2 [ 2378.283398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.741448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.822896] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2379.701369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2379.718603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.727472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.744992] LNet: Removed LNI 10.1.2.3@tcp [ 2380.754885] LNet: Removed LNI 192.168.201.51@tcp [ 2380.777857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.834975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2380.868088] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2380.875865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2381.490125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2381.530405] LNet: Removed LNI 192.168.201.51@tcp [ 2381.551360] LNet: Removed LNI 10.1.2.3@tcp [ 2388.503361] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:17:47 (1784189867) [ 2388.762554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.826173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.988219] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2389.132104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2389.144503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2389.162961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2391.811653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.868376] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2392.808370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2392.843044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2395.393463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2395.464921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2395.528330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.51@tcp -m PUT -e local_timeout -r 1 [ 2395.623727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2395.746328] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2395.748839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2396.099065] LNetError: 119534:0:(lib-move.c:4771:lnet_parse()) 192.168.201.51@tcp, src 192.168.201.51@tcp: Dropping PUT (error -108 looking up sender) [ 2397.161158] LNet: Removed LNI 10.1.2.3@tcp [ 2398.186711] LNet: Removed LNI 192.168.201.51@tcp [ 2398.766250] Key type .llcrypt unregistered [ 2398.775307] Key type ._llcrypt unregistered [ 2412.572436] Key type ._llcrypt registered [ 2412.576103] Key type .llcrypt registered [ 2413.904108] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:18:12 (1784189892) [ 2414.545356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.553139] alg: No test for adler32 (adler32-zlib) [ 2415.484856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.812565] LNet: 120848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.456412] Key type .llcrypt unregistered [ 2416.458110] Key type ._llcrypt unregistered [ 2423.102668] Key type ._llcrypt registered [ 2423.110034] Key type .llcrypt registered [ 2423.609458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2423.639516] alg: No test for adler32 (adler32-zlib) [ 2424.567443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2424.603419] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2433.322584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2433.605296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2434.353930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.151@tcp [ 2434.677143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2434.712285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2434.758926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2434.795664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2434.846807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2434.848620] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 2434.894612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2434.933856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2434.981308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.017438] LNet: 121298:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2435.053075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2435.086898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2435.119115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.145814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2435.185420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2435.235820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2436.082452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2436.351847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.410872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2436.467771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2436.527631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2436.563859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2436.624779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2436.668422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2436.717028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2436.738459] LNet: 121416:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2436.769695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2436.832168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2436.889822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2436.922812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2436.958762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2437.033539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2437.889792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.151@tcp [ 2438.150372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2438.185185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2438.235824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2438.294561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2438.351744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2438.399071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2438.445389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2438.488098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2438.508889] LNet: 121533:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2438.540705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2438.600176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2438.636712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2438.670565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2438.725762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2438.795805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2439.390609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2439.570500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2439.593645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2439.624721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2439.651180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2439.677746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2439.704784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2439.732621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2439.759768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2439.792273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2439.818336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2439.851452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2439.877840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2439.903207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2439.934997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2440.256410] LNet: 121697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.262800] LNetError: 121697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.278810] LNet: Removed LNI 192.168.201.51@tcp [ 2440.677547] Key type .llcrypt unregistered [ 2440.680462] Key type ._llcrypt unregistered [ 2458.611775] Key type ._llcrypt registered [ 2458.615263] Key type .llcrypt registered [ 2459.864879] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:18:58 (1784189938) [ 2460.300572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.319857] alg: No test for adler32 (adler32-zlib) [ 2461.241416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.586668] LNet: 122585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.212434] Key type .llcrypt unregistered [ 2462.215592] Key type ._llcrypt unregistered [ 2469.226839] Key type ._llcrypt registered [ 2469.228828] Key type .llcrypt registered [ 2469.450666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2469.466321] alg: No test for adler32 (adler32-zlib) [ 2470.362986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2470.385659] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2477.631971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2486.865199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.894268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2486.922887] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 2487.605099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2488.203916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.151@tcp [ 2488.445995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2488.486264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2488.522886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2488.555531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2488.577301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2488.605368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2488.641961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2488.680886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2488.704903] LNet: 123237:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2488.728354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2488.758525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2488.798916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2488.831697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2488.857606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2488.864287] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 2488.886791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2489.512439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2489.710326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2489.735588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2489.787489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2489.829896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2489.867384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2489.894116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2489.917327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2489.937281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2489.950972] LNet: 123354:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2489.982433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2490.021455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2490.072795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2490.103248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2490.161274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2490.196436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2490.909883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.151@tcp [ 2491.123817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2491.142966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2491.165411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2491.186811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2491.211256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2491.238497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2491.268603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2491.296732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2491.311232] LNet: 123471:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2491.332885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2491.371617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2491.418298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2491.459799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2491.495591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2491.535743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2492.183953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2492.384549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2492.405657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2492.429789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2492.454879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2492.486603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2492.513467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2492.549715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2492.574231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2492.595594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2492.619991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2492.660952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2492.693518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2492.748433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2492.803590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2498.147719] LNet: 123670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.156801] LNetError: 123670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.170245] LNet: Removed LNI 192.168.201.51@tcp [ 2498.192253] LNet: Removed LNI 192.168.201.51@tcp1 [ 2498.776091] Key type .llcrypt unregistered [ 2498.782277] Key type ._llcrypt unregistered [ 2517.040832] Key type ._llcrypt registered [ 2517.044828] Key type .llcrypt registered [ 2518.252768] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 04:19:57 (1784189997) [ 2520.149756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2522.268035] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:20:00 (1784190000) [ 2529.803331] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:20:09 (1784190009) [ 2538.138625] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:20:16 (1784190016) [ 2545.897901] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:20:24 (1784190024) [ 2554.873396] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:20:33 (1784190033) [ 2562.467843] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:20:41 (1784190041) [ 2570.609964] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:20:49 (1784190049) [ 2578.682749] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:20:57 (1784190057) [ 2586.650621] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:21:05 (1784190065) [ 2594.712615] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:21:13 (1784190073) [ 2603.039880] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:21:21 (1784190081) [ 2610.040645] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:21:29 (1784190089) [ 2617.457757] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:21:36 (1784190096) [ 2624.919614] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:21:43 (1784190103) [ 2631.883642] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:21:50 (1784190110) [ 2638.750049] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:21:57 (1784190117) [ 2645.928705] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:22:04 (1784190124) [ 2653.640928] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:22:12 (1784190132) [ 2654.152583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.167621] alg: No test for adler32 (adler32-zlib) [ 2655.180960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2655.343574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2655.375574] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2656.642365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2656.659982] LNet: 134669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.669455] LNetError: 134669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.700276] LNet: Removed LNI 192.168.201.51@tcp [ 2656.736035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2656.822981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2656.846843] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2664.397719] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:22:23 (1784190143) [ 2672.549233] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:22:31 (1784190151) [ 2681.066653] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:22:39 (1784190159) [ 2689.693633] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:22:48 (1784190168) [ 2697.659894] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:22:56 (1784190176) [ 2705.348315] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:23:04 (1784190184) [ 2712.836692] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:23:11 (1784190191) [ 2720.185561] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:23:19 (1784190199) [ 2727.339230] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:23:26 (1784190206) [ 2734.268151] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:23:33 (1784190213) [ 2741.449872] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:23:40 (1784190220) [ 2748.117991] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:23:47 (1784190227) [ 2754.677086] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2526 sec ========== 04:23:53 (1784190233) [ 2754.984817] LNet: 142136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2754.988975] LNetError: 142136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.009752] LNet: Removed LNI 192.168.201.51@tcp [ 2755.544617] Key type .llcrypt unregistered [ 2755.546530] Key type ._llcrypt unregistered [ 2769.233988] Key type ._llcrypt registered [ 2769.247212] Key type .llcrypt registered [ 2769.804930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2769.833938] alg: No test for adler32 (adler32-zlib) [ 2771.057528] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2771.251278] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2772.895379] Key type lgssc registered [ 2775.011153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2844.183966] Lustre: Mounted lustre-client [ 2849.571147] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2858.511686] Lustre: Unmounted lustre-client [ 2882.889686] Key type lgssc unregistered [ 2883.189646] LNet: 145065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2883.198179] LNetError: 145065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2883.225372] LNet: Removed LNI 192.168.201.51@tcp [ 2884.208218] Key type .llcrypt unregistered [ 2884.210347] Key type ._llcrypt unregistered