[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 420723722 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003269] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008322] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012130] LSM: Security Framework initializing [ 0.014044] Yama: becoming mindful. [ 0.015036] SELinux: Initializing. [ 0.016080] *** VALIDATE selinux *** [ 0.024737] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028856] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032057] *** VALIDATE tmpfs *** [ 0.033456] *** VALIDATE proc *** [ 0.035188] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038117] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.043008] Speculative Store Bypass: Vulnerable [ 0.046593] debug: unmapping init [mem 0xffffffff9d459000-0xffffffff9d460fff] [ 0.048897] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049738] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050021] ... version: 2 [ 0.051016] ... bit width: 48 [ 0.052010] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054014] ... max period: 00007fffffffffff [ 0.055016] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057294] rcu: Hierarchical SRCU implementation. [ 0.059764] smp: Bringing up secondary CPUs ... [ 0.060664] x86: Booting SMP configuration: [ 0.061034] .... node #0, CPUs: #1 #2 #3 [ 0.064781] smp: Brought up 1 node, 4 CPUs [ 0.067021] smpboot: Max logical packages: 1 [ 0.068028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.217902] node 0 deferred pages initialised in 145ms [ 0.221145] devtmpfs: initialized [ 0.222286] x86/mm: Memory block size: 128MB [ 0.225027] gcov: version magic: 0x41383552 [ 0.227443] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.230092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.232311] pinctrl core: initialized pinctrl subsystem [ 0.234147] [ 0.234547] ************************************************************* [ 0.236012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238013] ** ** [ 0.239010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.241011] ** ** [ 0.243012] ** This means that this kernel is built to expose internal ** [ 0.244009] ** IOMMU data structures, which may compromise security on ** [ 0.246011] ** your system. ** [ 0.248010] ** ** [ 0.250010] ** If you see this message and you are not debugging the ** [ 0.251008] ** kernel, report this immediately to your vendor! ** [ 0.253011] ** ** [ 0.255011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257011] ************************************************************* [ 0.259561] NET: Registered protocol family 16 [ 0.260365] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.264076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.266061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.271099] cpuidle: using governor menu [ 0.272725] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275021] PCI: Using configuration type 1 for base access [ 0.276129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.286026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288065] cryptd: max_cpu_qlen set to 1000 [ 0.291259] ACPI: Added _OSI(Module Device) [ 0.293017] ACPI: Added _OSI(Processor Device) [ 0.295020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.296012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.300836] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305453] ACPI: Interpreter enabled [ 0.307038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308013] ACPI: Using IOAPIC for interrupt routing [ 0.310114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313564] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.322987] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.336549] acpiphp: Slot [2] registered [ 0.337192] acpiphp: Slot [5] registered [ 0.338081] acpiphp: Slot [6] registered [ 0.339086] acpiphp: Slot [3] registered [ 0.340092] acpiphp: Slot [4] registered [ 0.342079] acpiphp: Slot [7] registered [ 0.343106] acpiphp: Slot [8] registered [ 0.345132] acpiphp: Slot [9] registered [ 0.346098] acpiphp: Slot [10] registered [ 0.348110] acpiphp: Slot [11] registered [ 0.349106] acpiphp: Slot [12] registered [ 0.351095] acpiphp: Slot [13] registered [ 0.352128] acpiphp: Slot [14] registered [ 0.354136] acpiphp: Slot [15] registered [ 0.355095] acpiphp: Slot [16] registered [ 0.357111] acpiphp: Slot [17] registered [ 0.359111] acpiphp: Slot [18] registered [ 0.360099] acpiphp: Slot [19] registered [ 0.361101] acpiphp: Slot [20] registered [ 0.362098] acpiphp: Slot [21] registered [ 0.364169] acpiphp: Slot [22] registered [ 0.365118] acpiphp: Slot [23] registered [ 0.367190] acpiphp: Slot [24] registered [ 0.368076] acpiphp: Slot [25] registered [ 0.369099] acpiphp: Slot [26] registered [ 0.371099] acpiphp: Slot [27] registered [ 0.372161] acpiphp: Slot [28] registered [ 0.374105] acpiphp: Slot [29] registered [ 0.375131] acpiphp: Slot [30] registered [ 0.377104] acpiphp: Slot [31] registered [ 0.378220] PCI host bridge to bus 0000:00 [ 0.379020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.382027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.384019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.386025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.388025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.391025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.393161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.395982] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.400136] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.412065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.416024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.418018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.421021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.423484] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.424636] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.428046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.430882] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434854] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.444029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.450787] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.455581] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.464017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.490019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.499104] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.505015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.510018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536206] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.539607] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.543189] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.546458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.549298] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.556056] iommu: Default domain type: Passthrough [ 0.558512] SCSI subsystem initialized [ 0.560185] ACPI: bus type USB registered [ 0.562154] usbcore: registered new interface driver usbfs [ 0.564077] usbcore: registered new interface driver hub [ 0.566100] usbcore: registered new device driver usb [ 0.567168] pps_core: LinuxPPS API ver. 1 registered [ 0.569013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.572058] PTP clock support registered [ 0.574137] EDAC MC: Ver: 3.0.0 [ 0.576145] PCI: Using ACPI for IRQ routing [ 0.578661] NetLabel: Initializing [ 0.579012] NetLabel: domain hash size = 128 [ 0.581014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.583094] NetLabel: unlabeled traffic allowed by default [ 0.585149] vgaarb: loaded [ 0.587287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.589016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.600016] clocksource: Switched to clocksource kvm-clock [ 0.712062] VFS: Disk quotas dquot_6.6.0 [ 0.713298] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.715686] *** VALIDATE ramfs *** [ 0.717020] *** VALIDATE hugetlbfs *** [ 0.718480] pnp: PnP ACPI init [ 0.720766] pnp: PnP ACPI: found 6 devices [ 0.741296] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.744822] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.746809] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.749105] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.751554] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.753996] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.756876] NET: Registered protocol family 2 [ 0.760066] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.765565] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.768989] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.775054] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.778740] TCP: Hash tables configured (established 65536 bind 65536) [ 0.782098] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.785445] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.788485] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.791618] NET: Registered protocol family 1 [ 0.794350] RPC: Registered named UNIX socket transport module. [ 0.796909] RPC: Registered udp transport module. [ 0.798522] RPC: Registered tcp transport module. [ 0.800248] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.802400] NET: Registered protocol family 44 [ 0.803939] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.805894] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.808220] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.810382] PCI: CLS 0 bytes, default 64 [ 0.811953] Unpacking initramfs... [ 2.250948] debug: unmapping init [mem 0xffff9869fcc64000-0xffff9869fffcffff] [ 2.255255] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.257529] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.260315] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.770900] Initialise system trusted keyrings [ 2.772740] Key type blacklist registered [ 2.774705] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.783486] zbud: loaded [ 2.786636] *** VALIDATE nfs *** [ 2.788017] *** VALIDATE nfs4 *** [ 2.789900] pstore: using deflate compression [ 2.795054] Platform Keyring initialized [ 2.902721] NET: Registered protocol family 38 [ 2.904480] Key type asymmetric registered [ 2.905993] Asymmetric key parser 'x509' registered [ 2.907761] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.910963] io scheduler mq-deadline registered [ 2.912939] io scheduler kyber registered [ 2.914512] io scheduler bfq registered [ 2.916298] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.918979] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.923212] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.926256] ACPI: Power Button [PWRF] [ 2.932275] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.938902] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.950529] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.979783] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.008951] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.016127] Non-volatile memory driver v1.3 [ 3.018274] Linux agpgart interface v0.103 [ 3.049148] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.052112] vda: detected capacity change from 0 to 74829824 [ 3.066933] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.070031] vdb: detected capacity change from 0 to 1073741824 [ 3.077447] libphy: Fixed MDIO Bus: probed [ 3.092383] usbcore: registered new interface driver usbserial_generic [ 3.095282] usbserial: USB Serial support registered for generic [ 3.098022] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.102344] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.104296] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.106823] mousedev: PS/2 mouse device common for all mice [ 3.110124] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.112628] rtc_cmos 00:05: RTC can wake from S4 [ 3.116987] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.121205] rtc_cmos 00:05: registered as rtc0 [ 3.121251] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.123248] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.129390] intel_pstate: CPU model not supported [ 3.133209] hid: raw HID events driver (C) Jiri Kosina [ 3.135665] usbcore: registered new interface driver usbhid [ 3.137912] usbhid: USB HID core driver [ 3.139660] drop_monitor: Initializing network drop monitor service [ 3.142282] Initializing XFRM netlink socket [ 3.144857] NET: Registered protocol family 10 [ 3.148513] Segment Routing with IPv6 [ 3.150072] NET: Registered protocol family 17 [ 3.152227] mpls_gso: MPLS GSO support [ 3.157071] RAS: Correctable Errors collector initialized. [ 3.159461] AVX version of gcm_enc/dec engaged. [ 3.161339] AES CTR mode by8 optimization enabled [ 3.240747] sched_clock: Marking stable (3240717127, 0)->(4129340157, -888623030) [ 3.255666] registered taskstats version 1 [ 3.257289] Loading compiled-in X.509 certificates [ 3.259935] zswap: loaded using pool lzo/zbud [ 3.291575] Key type big_key registered [ 3.302689] Key type encrypted registered [ 3.304463] ima: No TPM chip found, activating TPM-bypass! [ 3.306442] ima: Allocated hash algorithm: sha1 [ 3.308255] ima: No architecture policies found [ 3.309696] evm: Initialising EVM extended attributes: [ 3.311223] evm: security.selinux [ 3.312537] evm: security.ima [ 3.313725] evm: security.capability [ 3.315096] evm: HMAC attrs: 0x1 [ 3.317446] rtc_cmos 00:05: setting system clock to 2026-08-24 11:47:38 UTC (1787572058) [ 3.322724] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 3.326297] debug: unmapping init [mem 0xffffffff9d182000-0xffffffff9d458fff] [ 3.334115] Write protecting the kernel read-only data: 28672k [ 3.337315] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.340357] debug: unmapping init [mem 0xffffffff9c114000-0xffffffff9c1fffff] [ 3.377494] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.386305] systemd[1]: Detected virtualization kvm. [ 3.388327] systemd[1]: Detected architecture x86-64. [ 3.390081] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.413984] systemd[1]: No hostname configured. [ 3.416094] systemd[1]: Set hostname to . [ 3.418309] random: systemd: uninitialized urandom read (16 bytes read) [ 3.420705] systemd[1]: Initializing machine ID from random generator. [ 3.469870] random: ln: uninitialized urandom read (6 bytes read) [ 3.565943] random: systemd: uninitialized urandom read (16 bytes read) [ 3.568680] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.573337] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.579778] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.233250] device-mapper: uevent: version 1.0.3 [ 4.235689] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.978313] virtio_net virtio0 ens2: renamed from eth0 [ 5.027582] scsi host0: ata_piix [ 5.036630] scsi host1: ata_piix [ 5.038074] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.040704] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.714383] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.821170] random: crng init done [ 9.822802] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.215956] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.856468] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.603420] SELinux: Disabled at runtime. [ 12.759851] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.779074] systemd[1]: Detected virtualization kvm. [ 12.785951] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.152317] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.168439] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.183744] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.190739] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.196483] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.210330] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.230450] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ 14.394852] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 15.404828] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.155052] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.199616] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.841220] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.079443] EDAC sbridge: Ver: 1.1.2 [ 19.182540] Key type dns_resolver registered [ 19.604523] NFS: Registering the id_resolver key type [ 19.607108] Key type id_resolver registered [ 19.610942] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg609-client login: [ 93.330695] libcfs: loading out-of-tree module taints kernel. [ 93.741191] Key type ._llcrypt registered [ 93.746191] Key type .llcrypt registered [ 94.111483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 94.132420] alg: No test for adler32 (adler32-zlib) [ 94.336686] hrtimer: interrupt took 11283232 ns [ 95.689192] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 97.022932] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 98.903140] Key type lgssc registered [ 101.549790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 230.389777] Lustre: Mounted lustre-client - version 2.17.57_80_g9b8a413 [ 235.275785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.694590] Lustre: DEBUG MARKER: oleg609-client.virtnet: executing check_logdir /tmp/testlogs/ [ 253.311754] Lustre: DEBUG MARKER: oleg609-client.virtnet: executing yml_node [ 255.969936] Lustre: lustre-OST0000-osc-ffff986a47d0d800: disconnect after 24s idle [ 258.217236] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 260.993982] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 263.992217] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 266.001536] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 07:51:59 EDT 2026 [ 281.374098] Lustre: Unmounted lustre-client [ 310.960087] Key type lgssc unregistered [ 311.222237] LNet: 8429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.232261] LNetError: 8429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.257655] LNet: Removed LNI 192.168.206.9@tcp [ 312.147153] Key type .llcrypt unregistered [ 312.151680] Key type ._llcrypt unregistered [ 336.130467] Key type ._llcrypt registered [ 336.132273] Key type .llcrypt registered [ 336.389114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 336.415617] alg: No test for adler32 (adler32-zlib) [ 337.330905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 337.408501] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 337.426308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 341.741260] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 341.839802] LNet: 9308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.846110] LNetError: 9308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 341.871490] LNet: Removed LNI 192.168.206.9@tcp [ 342.792840] Key type .llcrypt unregistered [ 342.795826] Key type ._llcrypt unregistered [ 349.677811] Key type ._llcrypt registered [ 349.680749] Key type .llcrypt registered [ 351.102072] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:53:24 (1787572404) [ 351.386325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.403785] alg: No test for adler32 (adler32-zlib) [ 352.318629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.427816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.452138] LNet: 9733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.494192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.553794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.569395] LNet: 9740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 358.870256] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:53:32 (1787572412) [ 359.095084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.118544] LNet: 10318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.140707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.250048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 365.909831] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:53:39 (1787572419) [ 366.127768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.154339] LNet: 10905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.175956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.336211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 372.388331] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:53:46 (1787572426) [ 372.650377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.675066] LNet: 11489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.692611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.762755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 379.615368] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:53:53 (1787572433) [ 379.826449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.844570] LNet: 12074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.862428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.954677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 380.444369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 386.755401] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:54:00 (1787572440) [ 386.862550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.912260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.994766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 387.347695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 387.804753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 393.793563] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:54:07 (1787572447) [ 394.006062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.038288] LNet: 13262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.048350] LNet: 13262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.074611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.142232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 400.132569] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:54:14 (1787572454) [ 400.395728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.428355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.476759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 400.497724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 400.723772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 400.748534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.068594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 401.105107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.326738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 401.354542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.702463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 401.743788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.084835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 402.120987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.518927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 402.561770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.984272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 403.002505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.413713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 403.475731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.833659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 403.859813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.142505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 404.170525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.564553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 404.605764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.888602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 404.926117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.472658] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:54:25 (1787572465) [ 411.919527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.937684] LNet: 14527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.942337] LNet: 14527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 411.958120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.029661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.074575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 419.599638] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:54:32 (1787572472) [ 419.837577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.891127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.025277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.117892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.205759] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:54:40 (1787572480) [ 427.493784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.540336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.629659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 427.656771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 433.912205] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:54:47 (1787572487) [ 434.300303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.340791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.417604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 434.481133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 440.992150] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:54:54 (1787572494) [ 441.132161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.172317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.263556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 441.324157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 447.743195] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:55:01 (1787572501) [ 447.878381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.895755] LNet: 17455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.902750] LNet: 17455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 447.926273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.018697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 448.074778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 454.431406] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:55:08 (1787572508) [ 454.564452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.618976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.706721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.171186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.751975] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:55:15 (1787572515) [ 462.076284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.139959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.374830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.714654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.008736] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:55:23 (1787572523) [ 470.260706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.341560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.516423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.708378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.217195] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:55:31 (1787572531) [ 477.473890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.512928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.051864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.264723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.668445] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:55:38 (1787572538) [ 484.862578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.919546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.019378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.071323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.540913] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:55:45 (1787572545) [ 491.738026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.812755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.889889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.948302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.361360] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:55:53 (1787572553) [ 499.805055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.886599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.977600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 506.164299] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:56:00 (1787572560) [ 506.382887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.437937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.522745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 512.903687] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:56:06 (1787572566) [ 513.035359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.077655] LNet: 22893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.086553] LNet: 22893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 513.113946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.224838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 513.660251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 520.389672] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:56:14 (1787572574) [ 520.678019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.753948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.858590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 520.896930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 528.028967] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:56:21 (1787572581) [ 528.529675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.566319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.648693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 528.684677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 535.382711] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:56:29 (1787572589) [ 535.747920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.789316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.914312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 535.975544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 542.555667] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:56:36 (1787572596) [ 542.702431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.771448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.919954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 542.967104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 549.527403] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:56:43 (1787572603) [ 549.762941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.789564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.839339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 549.917570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 549.954995] LNetError: 25845:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 550.002564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 550.032458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 556.136329] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:56:50 (1787572610) [ 556.538587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.588979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.711404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6DG3cjestK [ 556.780917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6DG3cjestK [ 556.846673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.6DG3cjestK [ 563.532320] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:56:57 (1787572617) [ 563.869933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.903566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.948540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 563.976350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 564.014622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 570.021527] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:57:03 (1787572623) [ 570.339827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.386301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.486232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.536393] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 571.872762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 573.560313] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 575.631187] Key type lgssc registered [ 577.044687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 581.118366] LNetError: 27606:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.9@tcp99 has no available nets [ 584.134574] Key type lgssc unregistered [ 584.490019] LNetError: 28718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.521947] LNet: Removed LNI 192.168.206.9@tcp [ 585.325537] Key type .llcrypt unregistered [ 585.335330] Key type ._llcrypt unregistered [ 592.189381] Key type ._llcrypt registered [ 592.193431] Key type .llcrypt registered [ 593.503280] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:57:27 (1787572647) [ 594.366977] Key type .llcrypt unregistered [ 594.374705] Key type ._llcrypt unregistered [ 601.253556] Key type ._llcrypt registered [ 601.255657] Key type .llcrypt registered [ 601.414929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 601.433727] alg: No test for adler32 (adler32-zlib) [ 602.383907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.497855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 602.521774] LNet: 29640:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 602.532930] LNet: 29640:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 602.541085] LNet: 29640:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 602.553741] LNet: Message forwarding will be enabled in 2s [ 605.663114] LNet: Message forwarding enabled [ 606.069386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 606.103182] LNet: Message forwarding will stop in 2s to 6s [ 609.764372] LNet: Message forwarding disabled [ 610.714400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 610.746244] LNet: 29739:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 610.756482] LNet: 29739:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 610.767749] LNet: 29739:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 610.792292] LNet: Message forwarding will be enabled in 2s [ 611.000273] LNet: 29752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.559812] Key type .llcrypt unregistered [ 611.562890] Key type ._llcrypt unregistered [ 623.150227] Key type ._llcrypt registered [ 623.152687] Key type .llcrypt registered [ 624.528528] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:57:58 (1787572678) [ 624.839399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.857112] alg: No test for adler32 (adler32-zlib) [ 625.851308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.957431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 631.741883] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:58:05 (1787572685) [ 631.878312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.910458] LNet: 31156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.952595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.998018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 632.025546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 632.055264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 632.080656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 632.106477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 632.134218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 632.164592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 632.194209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 632.253239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 632.323138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 632.360950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 632.435113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 632.481889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 632.522210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 632.555403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 632.585631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 632.607860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 632.638339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 632.669513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 632.692312] LNetError: 31202:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 639.839095] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:58:13 (1787572693) [ 640.052541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.087520] LNet: 31774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.142166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.245274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.439263] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:58:20 (1787572700) [ 649.079561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg609-server [ 651.522276] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:58:25 (1787572705) [ 651.659457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.691361] LNet: 32698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.716189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.755858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.776709] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 652.675919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[10-14]@tcp [ 652.690243] LNet: 32838:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 652.730055] LNetError: 32707:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.10@tcp is being used as a gateway but routing feature is not turned on [ 652.750085] LNetError: 32707:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.10@tcp has gone from up to down [ 660.464884] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:58:34 (1787572714) [ 662.020291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg609-server [ 663.979484] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:58:37 (1787572717) [ 664.299767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.328777] LNet: 33774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.337963] LNetError: 33774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.354166] LNet: Removed LNI 192.168.206.9@tcp [ 664.390994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.459087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.478791] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 665.114653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[10-15/2]@tcp [ 665.136447] LNet: 33891:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 665.142117] LNet: 33891:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 665.163281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[10-15/2]@tcp [ 665.164411] LNetError: 33778:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.14@tcp is being used as a gateway but routing feature is not turned on [ 665.202108] LNetError: 33778:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 665.207153] LNetError: 33778:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.14@tcp has gone from up to down [ 665.213380] LNetError: 33778:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 671.742346] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:58:45 (1787572725) [ 672.032431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.061400] LNet: 34483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.065350] LNetError: 34483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.083160] LNet: Removed LNI 192.168.206.9@tcp [ 672.113513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.206583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 672.277781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.338128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.416361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.500213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.563185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.635436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 672.690847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.739596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.796458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.911896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.962986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.007408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 673.069178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.103213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.150522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.220729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.251608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.306885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 673.401219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.451651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.514660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.574470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.601607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.639173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 673.692716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.718297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.774414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.863089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.885459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.917350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 673.955154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.979324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.054957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.742927] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:58:53 (1787572733) [ 681.335983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg609-server [ 682.779681] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:58:56 (1787572736) [ 685.015887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg609-server [ 686.180139] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:59:00 (1787572740) [ 686.384543] LNet: 35885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.388235] LNet: 35885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 686.905339] Key type .llcrypt unregistered [ 686.907850] Key type ._llcrypt unregistered [ 692.413887] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 692.537553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 692.544078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 692.555126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 695.095675] Key type ._llcrypt registered [ 695.097133] Key type .llcrypt registered [ 695.252181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.262670] alg: No test for adler32 (adler32-zlib) [ 696.281548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.360347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.400243] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 697.350928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.382265] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.015186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 700.053582] LNet: Removed LNI 10.1.2.3@tcp [ 700.290801] LNet: 36603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.308022] LNetError: 36603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.342038] LNet: Removed LNI 192.168.206.9@tcp [ 701.195659] Key type .llcrypt unregistered [ 701.198794] Key type ._llcrypt unregistered [ 715.851451] Key type ._llcrypt registered [ 715.853626] Key type .llcrypt registered [ 717.275182] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:59:30 (1787572770) [ 718.059284] Key type .llcrypt unregistered [ 718.065255] Key type ._llcrypt unregistered [ 724.103585] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.237417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.246720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.252441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.765332] Key type ._llcrypt registered [ 726.769162] Key type .llcrypt registered [ 726.973169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.985861] alg: No test for adler32 (adler32-zlib) [ 727.894511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.959521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.982969] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 728.623531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 728.637063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 731.483467] LNet: 38101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.495335] LNetError: 38101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 731.505443] LNet: Removed LNI 10.1.2.3@tcp [ 731.512624] LNet: Removed LNI 192.168.206.9@tcp [ 732.213624] Key type .llcrypt unregistered [ 732.218507] Key type ._llcrypt unregistered [ 744.341136] Key type ._llcrypt registered [ 744.342759] Key type .llcrypt registered [ 745.615659] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:59:59 (1787572799) [ 746.155337] Key type .llcrypt unregistered [ 746.159205] Key type ._llcrypt unregistered [ 752.431294] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.541230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.551317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.556759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.328274] Key type ._llcrypt registered [ 755.330965] Key type .llcrypt registered [ 755.525827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.549927] alg: No test for adler32 (adler32-zlib) [ 756.437965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.484814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 756.510304] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 758.399239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.430811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.556927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 760.584837] LNet: Removed LNI 10.1.2.3@tcp [ 760.619141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 760.660243] LNet: Removed LNI 10.1.2.31@tcp [ 760.695793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.730128] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.648544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 761.684490] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 762.736456] LNet: 40030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.740920] LNetError: 40030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 762.745619] LNet: Removed LNI 10.1.2.31@tcp [ 762.753501] LNet: Removed LNI 10.1.2.3@tcp [ 763.340352] Key type .llcrypt unregistered [ 763.341820] Key type ._llcrypt unregistered [ 775.458190] Key type ._llcrypt registered [ 775.465306] Key type .llcrypt registered [ 776.885379] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:00:30 (1787572830) [ 777.293663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.312620] alg: No test for adler32 (adler32-zlib) [ 778.221296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.281922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.326038] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 779.048161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.075574] LNet: 40988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.087454] LNetError: 40988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.101508] LNet: Removed LNI 192.168.206.9@tcp [ 779.112659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.163846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.9@tcp [ 779.192729] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 779.728075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.9@tcp [ 779.761111] LNet: Removed LNI 192.168.206.9@tcp [ 779.805322] LNet: 41123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.456367] Key type .llcrypt unregistered [ 780.460309] Key type ._llcrypt unregistered [ 791.649588] Key type ._llcrypt registered [ 791.654737] Key type .llcrypt registered [ 792.859423] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:00:46 (1787572846) [ 794.430697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg609-server [ 796.463849] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:00:50 (1787572850) [ 797.122109] Key type .llcrypt unregistered [ 797.123956] Key type ._llcrypt unregistered [ 803.238083] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.382471] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.394446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.401290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 806.123974] Key type ._llcrypt registered [ 806.127329] Key type .llcrypt registered [ 806.364838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.387692] alg: No test for adler32 (adler32-zlib) [ 807.340869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.392592] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 808.078752] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 809.883825] LNet: 42975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.896860] LNetError: 42975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.903552] LNet: Removed LNI 192.168.206.9@tcp1 [ 809.922653] LNet: Removed LNI 10.1.2.3@tcp2 [ 810.663574] Key type .llcrypt unregistered [ 810.675572] Key type ._llcrypt unregistered [ 820.683601] Key type ._llcrypt registered [ 820.685822] Key type .llcrypt registered [ 821.926828] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:01:15 (1787572875) [ 822.222405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.243872] alg: No test for adler32 (adler32-zlib) [ 823.130939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.182674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.200702] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 823.813374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.826838] LNet: 43949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.830659] LNetError: 43949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.848653] LNet: Removed LNI 192.168.206.9@tcp [ 823.889302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.944427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 823.973087] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 829.728771] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:01:23 (1787572883) [ 829.811038] LNet: 44531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.819892] LNetError: 44531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.839465] LNet: Removed LNI 192.168.206.9@tcp [ 830.481521] Key type .llcrypt unregistered [ 830.484625] Key type ._llcrypt unregistered [ 837.204903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.377722] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.382053] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.384487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.899275] Key type ._llcrypt registered [ 839.906406] Key type .llcrypt registered [ 840.066232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.083896] alg: No test for adler32 (adler32-zlib) [ 840.957804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.988778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.026036] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 841.635929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 841.655607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.547855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.568257] LNet: 45286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.574300] LNetError: 45286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.585453] LNet: Removed LNI 10.1.2.3@tcp [ 843.607125] LNet: Removed LNI 192.168.206.9@tcp [ 843.626560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.698900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 843.722347] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 843.730746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.086853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.107373] LNet: 45313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.114188] LNetError: 45313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 844.120271] LNetError: 45313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 844.124713] LNet: Removed LNI 10.1.2.3@tcp [ 844.135523] LNet: Removed LNI 192.168.206.9@tcp [ 844.155321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.207722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 844.225236] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 844.231158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.571840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 844.594673] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 845.554085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.587161] LNet: 45476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.596232] LNetError: 45476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 845.601474] LNetError: 45476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.607137] LNet: Removed LNI 10.1.2.3@tcp [ 845.612565] LNet: Removed LNI 192.168.206.9@tcp [ 845.623508] LNet: Removed LNI 192.168.206.9@tcp2 [ 845.637994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.690305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 845.716805] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 845.721538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.731563] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 851.502824] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:01:45 (1787572905) [ 851.697830] LNet: 46073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.702601] LNetError: 46073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.707629] LNetError: 46073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 851.712223] LNet: Removed LNI 10.1.2.3@tcp [ 851.716348] LNet: Removed LNI 192.168.206.9@tcp [ 851.730680] LNet: Removed LNI 192.168.206.9@tcp2 [ 852.303678] Key type .llcrypt unregistered [ 852.307341] Key type ._llcrypt unregistered [ 858.391037] Key type ._llcrypt registered [ 858.397496] Key type .llcrypt registered [ 858.562677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.575704] alg: No test for adler32 (adler32-zlib) [ 859.463174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.525140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.552069] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 860.257094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.271346] LNet: 46471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.276181] LNetError: 46471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.290230] LNet: Removed LNI 192.168.206.9@tcp [ 860.305564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.351221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 860.398877] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 860.829931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.866142] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.879771] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.901910] LNet: Removed LNI 192.168.206.9@tcp [ 860.927869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.007979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 861.039733] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 861.548144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.591095] LNet: Removed LNI 192.168.206.9@tcp [ 861.605429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.655246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 861.706308] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 862.227916] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 862.338786] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 862.341301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 862.347715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.676348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.708663] LNet: 46605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.726244] LNet: 46605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 864.736617] LNetError: 46605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.746107] LNetError: 46605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 864.808539] LNet: Removed LNI 192.168.206.9@tcp [ 864.830702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.875941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.895493] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 870.681432] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:02:04 (1787572924) [ 870.782255] LNet: 47193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.790977] LNetError: 47193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.809316] LNet: Removed LNI 192.168.206.9@tcp [ 871.412966] Key type .llcrypt unregistered [ 871.415972] Key type ._llcrypt unregistered [ 878.827908] Key type ._llcrypt registered [ 878.830351] Key type .llcrypt registered [ 879.034612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.052704] alg: No test for adler32 (adler32-zlib) [ 880.035989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.094969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.117789] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 881.044778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.068923] LNet: 47604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.080628] LNetError: 47604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.107331] LNet: Removed LNI 192.168.206.9@tcp [ 881.176345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.304199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 881.330382] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 888.284271] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:02:21 (1787572941) [ 888.412549] LNet: 48188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.434651] LNetError: 48188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.453572] LNet: Removed LNI 192.168.206.9@tcp [ 889.461835] Key type .llcrypt unregistered [ 889.465966] Key type ._llcrypt unregistered [ 895.286047] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.473628] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.478539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.489995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 897.990715] Key type ._llcrypt registered [ 897.999469] Key type .llcrypt registered [ 898.282433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.300493] alg: No test for adler32 (adler32-zlib) [ 899.198466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.250537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.281687] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 900.041247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.069987] LNet: 48651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.074555] LNetError: 48651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.096214] LNet: Removed LNI 192.168.206.9@tcp [ 900.122159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.164822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 900.206853] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 907.622262] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:02:41 (1787572961) [ 907.746801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.805198] LNet: 49246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.811927] LNetError: 49246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.841713] LNet: Removed LNI 192.168.206.9@tcp [ 907.868547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.005634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 914.349362] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:02:48 (1787572968) [ 914.585344] LNet: 49831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.277156] Key type .llcrypt unregistered [ 915.282820] Key type ._llcrypt unregistered [ 921.559186] Key type ._llcrypt registered [ 921.562083] Key type .llcrypt registered [ 921.863599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.898160] alg: No test for adler32 (adler32-zlib) [ 922.799456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.866165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.902505] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 923.773148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.801848] LNet: 50236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.810866] LNetError: 50236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.820834] LNet: Removed LNI 192.168.206.9@tcp [ 923.850605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.952392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 923.980805] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 929.858227] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:03:03 (1787572983) [ 930.055486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.089381] LNet: 50829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.095733] LNetError: 50829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.112279] LNet: Removed LNI 192.168.206.9@tcp [ 930.136929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.232562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 936.865592] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:03:10 (1787572990) [ 938.427930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 940.267352] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:03:13 (1787572993) [ 941.640914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 943.625206] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:03:17 (1787572997) [ 945.248732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 946.999684] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:03:20 (1787573000) [ 948.494836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 950.283285] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:03:24 (1787573004) [ 951.865086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 953.304076] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:03:27 (1787573007) [ 953.643300] LNet: 52882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.332896] Key type .llcrypt unregistered [ 954.340234] Key type ._llcrypt unregistered [ 961.051412] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 961.223393] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 961.238955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 961.249793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 963.805625] Key type ._llcrypt registered [ 963.809663] Key type .llcrypt registered [ 964.017753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.037945] alg: No test for adler32 (adler32-zlib) [ 964.996793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.081728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.110168] LNet: Added LNI 192.168.206.9@tcp [8/512/0/180] [ 965.120189] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 965.127879] LNet: 53206:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 965.239110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.266400] LNet: 53225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.273692] LNetError: 53225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.281760] LNet: Removed LNI 10.1.2.3@tcp [ 965.283405] LNetError: 53225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.295540] LNet: Removed LNI 192.168.206.9@tcp [ 965.331600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.390895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.421725] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 965.473665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.492080] LNet: 53249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.498521] LNetError: 53249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.511978] LNet: Removed LNI 192.168.206.9@tcp [ 965.540368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.612551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.650431] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 965.659186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.663468] LNet: 53257:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 965.746710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.765794] LNet: 53275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.775045] LNetError: 53275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.783897] LNet: Removed LNI 10.1.2.3@tcp [ 965.789850] LNetError: 53275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.808043] LNet: Removed LNI 192.168.206.9@tcp [ 965.831648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.893754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 965.916168] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 965.920115] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.922937] LNet: 53283:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 966.149471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.183921] LNet: 53309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.199512] LNetError: 53309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.217565] LNet: Removed LNI 10.1.2.3@tcp [ 966.220335] LNetError: 53309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.229562] LNet: Removed LNI 192.168.206.9@tcp [ 966.264630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.335647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 966.352516] LNet: Added LNI 192.168.206.9@tcp [8/256/0/182] [ 966.358905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 966.473162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.503407] LNet: 53337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.514147] LNetError: 53337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 966.521601] LNet: Removed LNI 10.1.2.3@tcp [ 966.523824] LNetError: 53337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.537586] LNet: Removed LNI 192.168.206.9@tcp [ 966.579632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.662155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 966.683974] LNet: Added LNI 192.168.206.9@tcp [8/256/0/182] [ 966.689197] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 966.690966] LNet: 53345:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 966.771701] LNet: Removed LNI 10.1.2.3@tcp [ 966.784486] LNet: Removed LNI 192.168.206.9@tcp [ 967.414737] Key type .llcrypt unregistered [ 967.416777] Key type ._llcrypt unregistered [ 978.332299] Key type ._llcrypt registered [ 978.334573] Key type .llcrypt registered [ 979.327656] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:03:53 (1787573033) [ 980.818352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 982.843552] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:03:56 (1787573036) [ 983.188060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.201207] alg: No test for adler32 (adler32-zlib) [ 984.117927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.182535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 984.202017] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 984.877817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.9 [ 984.899336] LNet: 54614:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 984.915563] LNet: Removed LNI 192.168.206.9@tcp [ 991.367735] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:04:04 (1787573044) [ 991.500623] LNet: 55177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.506768] LNetError: 55177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.524346] LNet: Removed LNI 192.168.206.9@tcp [ 992.349692] Key type .llcrypt unregistered [ 992.353659] Key type ._llcrypt unregistered [ 998.044594] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.220540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.905053] Key type ._llcrypt registered [ 1000.913265] Key type .llcrypt registered [ 1001.073275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.089219] alg: No test for adler32 (adler32-zlib) [ 1002.392449] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.557811] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:04:22 (1787573062) [ 1008.749489] LNet: 56104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.755775] LNetError: 56104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1008.776132] LNet: Removed LNI 10.1.2.3@tcp [ 1009.396194] Key type .llcrypt unregistered [ 1009.399152] Key type ._llcrypt unregistered [ 1015.612936] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1015.862439] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1015.872856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1015.883899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1017.606849] Key type ._llcrypt registered [ 1017.610932] Key type .llcrypt registered [ 1017.756038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.766675] alg: No test for adler32 (adler32-zlib) [ 1018.809625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1025.559746] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:04:39 (1787573079) [ 1026.994333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1028.916568] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:04:42 (1787573082) [ 1029.029246] LNet: 57274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.036736] LNetError: 57274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1029.058773] LNet: Removed LNI 10.1.2.3@tcp [ 1029.769687] Key type .llcrypt unregistered [ 1029.772105] Key type ._llcrypt unregistered [ 1036.749229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1036.946184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1036.949288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1036.961987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1039.634307] Key type ._llcrypt registered [ 1039.636510] Key type .llcrypt registered [ 1039.820058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1039.837911] alg: No test for adler32 (adler32-zlib) [ 1040.839721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.907344] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.187334] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:05:02 (1787573102) [ 1048.748155] LNet: 58516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.752655] LNetError: 58516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.766921] LNet: Removed LNI 10.1.2.3@tcp [ 1049.296750] Key type .llcrypt unregistered [ 1049.298538] Key type ._llcrypt unregistered [ 1054.193987] Key type ._llcrypt registered [ 1054.199885] Key type .llcrypt registered [ 1054.339943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.354689] alg: No test for adler32 (adler32-zlib) [ 1055.264431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1055.286148] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1062.240516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1062.863090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_interrupt -r 1 [ 1062.925553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_interrupt -r 1 [ 1062.955197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1062.972317] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1063.164847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.191246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.226543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.275467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.322130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.358739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.433199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.475950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.500935] LNet: 58958:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.517457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.519769] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1063.523880] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1063.550475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.574902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.600317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.634095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.658793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.684583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.186976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_dropped -r 1 [ 1064.221487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_dropped -r 1 [ 1064.249532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1064.440497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.463455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.483634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.508661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.533553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.560689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.581081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.600684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.612199] LNet: 59068:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.623849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.643699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.664658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.689896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.710587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.737502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.760113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1065.218583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_aborted -r 1 [ 1065.244534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_aborted -r 1 [ 1065.273741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1065.299939] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1065.308471] LNet: Skipped 1 previous similar message [ 1065.530324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.555716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.580823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.604512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.631702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.656956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.682789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.719945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.737610] LNet: 59177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.755673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.794806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.866940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.902753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.948910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.981828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.017632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.509426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_no_route -r 1 [ 1066.561035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_no_route -r 1 [ 1066.621530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1066.840661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.877860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.936988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1066.976844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.024438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.056099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.096970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.130983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.166221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.207774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.251510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.306600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.352239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.396269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.435721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1067.957234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_timeout -r 1 [ 1068.000892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_timeout -r 1 [ 1068.046748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1068.067952] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1068.074984] LNet: Skipped 2 previous similar messages [ 1068.317783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.344981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.370144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.421437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.455687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.497467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.528840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.562587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.576942] LNet: 59395:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.581245] LNet: 59395:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1068.595191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.621270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.665199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.688746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.716165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.742540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.769937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1069.305099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_error -r 1 [ 1069.337946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_error -r 1 [ 1069.369733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1069.626994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.671470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.716767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.765931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.802688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.836614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.871875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.909147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.950818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.995252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.026771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.061941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.098676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.133452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.171374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1070.485558] LNet: 59550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.491663] LNetError: 59550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.530034] LNet: Removed LNI 192.168.206.9@tcp [ 1071.952491] Key type .llcrypt unregistered [ 1071.959873] Key type ._llcrypt unregistered [ 1088.341992] Key type ._llcrypt registered [ 1088.345536] Key type .llcrypt registered [ 1090.038659] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:05:43 (1787573143) [ 1091.058631] Key type .llcrypt unregistered [ 1091.062044] Key type ._llcrypt unregistered [ 1098.784825] Key type ._llcrypt registered [ 1098.788268] Key type .llcrypt registered [ 1098.941944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1098.952764] alg: No test for adler32 (adler32-zlib) [ 1099.857325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1099.882802] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1106.655969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1117.595694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.654097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1117.696274] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1118.914712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_interrupt -r 1 [ 1118.941813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 1 [ 1118.970657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_interrupt -r 1 [ 1119.004364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 1 [ 1119.033427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_interrupt -r 1 [ 1119.087352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 1 [ 1119.140600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_interrupt -r 1 [ 1119.187895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 1 [ 1119.231196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1119.248257] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1119.254553] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1119.438423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.476330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.521626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.567412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.598603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.624688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.652770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.678398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.694649] LNet: 61024:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.709735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.742783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.773725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.811327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.854657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.887245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.921964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.394793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_dropped -r 1 [ 1120.449713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 1 [ 1120.509932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_dropped -r 1 [ 1120.561583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 1 [ 1120.589296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_dropped -r 1 [ 1120.627726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 1 [ 1120.680623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_dropped -r 1 [ 1120.749541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 1 [ 1120.793835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1120.810647] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1120.814463] LNet: Skipped 2 previous similar messages [ 1120.987188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.015391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.041820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.073779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.094289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.118656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.147050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.176481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.190781] LNet: 61146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.208384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.230902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.258064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.280934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.303537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.334521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.371506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.809283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_aborted -r 1 [ 1121.834727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 1 [ 1121.864801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_aborted -r 1 [ 1121.917498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 1 [ 1121.967781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_aborted -r 1 [ 1122.003666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 1 [ 1122.031943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_aborted -r 1 [ 1122.076675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 1 [ 1122.122917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1122.139346] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1122.144720] LNet: Skipped 2 previous similar messages [ 1122.425368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.455671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.481764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.511047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.538691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.565472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.592813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.616666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.635652] LNet: 61267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.652561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.683309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.711827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.748873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.778836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.815836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.863854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.410695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_no_route -r 1 [ 1123.454589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 1 [ 1123.516131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_no_route -r 1 [ 1123.565536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 1 [ 1123.622268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_no_route -r 1 [ 1123.718423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 1 [ 1123.773756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_no_route -r 1 [ 1123.831931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 1 [ 1123.889968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1124.170758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.226723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.269140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.311800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.366955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.432652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.475240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.539262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.572357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.617717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.650756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.676905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.703722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.735829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.768807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.355649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_timeout -r 1 [ 1125.386964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 1 [ 1125.424415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_timeout -r 1 [ 1125.459977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 1 [ 1125.491427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_timeout -r 1 [ 1125.527672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 1 [ 1125.561255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_timeout -r 1 [ 1125.591718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 1 [ 1125.625764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1125.648374] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1125.653587] LNet: Skipped 5 previous similar messages [ 1125.870624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.899845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.929440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.962748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.002890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.037298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.066575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.100897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.121348] LNet: 61510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.131366] LNet: 61510:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1126.162996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.198562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.226364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.290517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.325684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.358798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.400912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.976121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_error -r 1 [ 1127.014251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_error -r 1 [ 1127.048326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_error -r 1 [ 1127.081376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_error -r 1 [ 1127.118736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_error -r 1 [ 1127.157661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_error -r 1 [ 1127.206608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_error -r 1 [ 1127.256900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_error -r 1 [ 1127.313539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1127.566116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.589361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.618836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.644280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.674739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.697787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.729843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.780362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.820373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.856357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.905220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.951466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.992373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.031290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.067748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.572564] LNet: 61711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.588865] LNetError: 61711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.601099] LNet: Removed LNI 192.168.206.9@tcp [ 1134.611809] LNet: Removed LNI 192.168.206.9@tcp1 [ 1134.982657] Key type .llcrypt unregistered [ 1134.987403] Key type ._llcrypt unregistered [ 1152.334410] Key type ._llcrypt registered [ 1152.336510] Key type .llcrypt registered [ 1153.745174] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:06:47 (1787573207) [ 1154.815320] Key type .llcrypt unregistered [ 1154.817389] Key type ._llcrypt unregistered [ 1161.895291] Key type ._llcrypt registered [ 1161.896895] Key type .llcrypt registered [ 1162.047064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.059812] alg: No test for adler32 (adler32-zlib) [ 1162.985485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1163.005345] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1170.475232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1171.072506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_dropped -r 1 [ 1171.120764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_dropped -r 1 [ 1171.172441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1171.197685] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1171.443445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.481994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.526376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.559753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.596597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.646957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.697463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.748357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.768486] LNet: 62966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.799965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.837591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.881165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.914498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.958447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.990367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.037436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.576212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_error -r 1 [ 1172.643075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_error -r 1 [ 1172.715463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1172.742870] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1173.023341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.048305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.108096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.168499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.217956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.258487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.313098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.359311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.389306] LNet: 63076:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.421755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.470872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.518591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.585818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.654897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.709321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.759371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.261516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_timeout -r 1 [ 1174.298294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_timeout -r 1 [ 1174.329660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1174.358044] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1174.649129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.684646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.732247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.762601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.792334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.821447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.853429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.878291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.891684] LNet: 63185:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.907462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.935612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.998626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.050656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.100081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.131366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.158740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.409269] LNet: 63230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.415635] LNetError: 63230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.438772] LNet: Removed LNI 192.168.206.9@tcp [ 1175.835467] Key type .llcrypt unregistered [ 1175.838629] Key type ._llcrypt unregistered [ 1192.389151] Key type ._llcrypt registered [ 1192.390658] Key type .llcrypt registered [ 1193.676653] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:07:27 (1787573247) [ 1194.824254] Key type .llcrypt unregistered [ 1194.826057] Key type ._llcrypt unregistered [ 1201.226841] Key type ._llcrypt registered [ 1201.230277] Key type .llcrypt registered [ 1201.473880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.490584] alg: No test for adler32 (adler32-zlib) [ 1202.451773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1202.477761] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1210.922524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1220.055957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.098197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1220.128349] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1221.703461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_dropped -r 1 [ 1221.736751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 1 [ 1221.772801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_dropped -r 1 [ 1221.805705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 1 [ 1221.837919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_dropped -r 1 [ 1221.869247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 1 [ 1221.913423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_dropped -r 1 [ 1221.964410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 1 [ 1222.015510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1222.043961] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1222.055437] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1222.325376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.361840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.405370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.452085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.499491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.535747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.598533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.639787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.665322] LNet: 64701:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.690874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.735961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.781149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.848394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.927418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.978641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.024486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1223.060926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.585431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_error -r 1 [ 1223.612656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_error -r 1 [ 1223.646948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_error -r 1 [ 1223.678099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_error -r 1 [ 1223.726833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_error -r 1 [ 1223.783865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_error -r 1 [ 1223.837627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_error -r 1 [ 1223.893721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_error -r 1 [ 1223.954967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1223.985613] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1223.997174] LNet: Skipped 2 previous similar messages [ 1224.231314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.260815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.300132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.332689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.365752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.392920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.427327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.454628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.474719] LNet: 64825:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.487175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.512304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.547779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.586530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.616157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.650866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.689749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.728308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.254692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_timeout -r 1 [ 1225.292437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_timeout -r 1 [ 1225.321174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_timeout -r 1 [ 1225.352446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_timeout -r 1 [ 1225.385020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_timeout -r 1 [ 1225.437138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_timeout -r 1 [ 1225.508152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_timeout -r 1 [ 1225.555072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_timeout -r 1 [ 1225.601529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 1225.618431] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1225.622204] LNet: Skipped 1 previous similar message [ 1225.850529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.876646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.903335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.932963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.962207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.984899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.047653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.085925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.108358] LNet: 64948:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.142737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.188946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.225695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.281030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.331352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.361987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.404774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1226.486138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.663469] LNet: 65030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.675168] LNetError: 65030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.683957] LNet: Removed LNI 192.168.206.9@tcp [ 1231.702398] LNet: Removed LNI 192.168.206.9@tcp1 [ 1232.222787] Key type .llcrypt unregistered [ 1232.228699] Key type ._llcrypt unregistered [ 1248.766552] Key type ._llcrypt registered [ 1248.768788] Key type .llcrypt registered [ 1250.151328] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:08:23 (1787573303) [ 1251.691574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1253.642327] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:08:27 (1787573307) [ 1254.816407] Key type .llcrypt unregistered [ 1254.823622] Key type ._llcrypt unregistered [ 1262.029729] Key type ._llcrypt registered [ 1262.032694] Key type .llcrypt registered [ 1262.262227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1262.292857] alg: No test for adler32 (adler32-zlib) [ 1263.239602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1263.283951] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1271.236549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1272.077201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e network_timeout -r 1 [ 1272.135358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e network_timeout -r 1 [ 1272.219855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1272.238707] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1272.510832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.542438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.543585] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1272.587756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.628975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.656322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.689462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.750715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1272.809854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1272.846589] LNet: 66578:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.897641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1272.968376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.038156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.085432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.124381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.164543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.545439] LNet: 66621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.565595] LNetError: 66621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.578149] LNet: Removed LNI 192.168.206.9@tcp [ 1274.054706] Key type .llcrypt unregistered [ 1274.062813] Key type ._llcrypt unregistered [ 1292.416515] Key type ._llcrypt registered [ 1292.419237] Key type .llcrypt registered [ 1292.594420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.608141] alg: No test for adler32 (adler32-zlib) [ 1293.453907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1293.474168] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1301.636250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1312.432466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.477653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.502327] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1313.589330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e network_timeout -r 1 [ 1313.637777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e network_timeout -r 1 [ 1313.685273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e network_timeout -r 1 [ 1313.731532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e network_timeout -r 1 [ 1313.777784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e network_timeout -r 1 [ 1313.834433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e network_timeout -r 1 [ 1313.930777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e network_timeout -r 1 [ 1313.999378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e network_timeout -r 1 [ 1314.061197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1314.090903] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1314.342715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.370717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1314.415524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1314.453247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1314.504887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1314.590078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1314.641630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1314.701362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1314.750268] LNet: 67506:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1314.795839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1314.854920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1314.907386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1314.961205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1314.975171] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1314.998903] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1315.035778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.079339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.927604] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1321.373989] LNet: 67585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.385487] LNetError: 67585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.390799] LNet: Removed LNI 192.168.206.9@tcp [ 1321.404354] LNet: Removed LNI 192.168.206.9@tcp1 [ 1321.818549] Key type .llcrypt unregistered [ 1321.823944] Key type ._llcrypt unregistered [ 1337.618822] Key type ._llcrypt registered [ 1337.621282] Key type .llcrypt registered [ 1338.614050] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:09:52 (1787573392) [ 1339.290790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.315655] alg: No test for adler32 (adler32-zlib) [ 1340.195135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.241348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.258389] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1340.951418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.987238] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1341.774245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1341.845894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1341.900930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1341.934565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1342.013283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1342.058620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1342.094323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.9@tcp [ 1342.116695] LNetError: 68685:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1342.303294] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1342.304367] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1344.358975] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1345.552965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1348.447587] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1349.034444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1349.070468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1349.138875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.160685] LNet: 68792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.165505] LNetError: 68792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.170350] LNet: Removed LNI 192.168.206.9@tcp [ 1349.177340] LNet: Removed LNI 192.168.206.9@tcp1 [ 1349.205539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.280928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.340227] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1350.022266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.041475] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1350.844557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1350.897579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.978064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1351.040277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1351.085566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1351.127603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1351.158329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1351.185194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.9@tcp [ 1351.203266] LNetError: 69065:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1351.327772] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1353.556252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1357.472571] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1357.482653] LNet: Skipped 1 previous similar message [ 1362.543516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1362.567252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1362.587141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.709339] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:10:22 (1787573422) [ 1368.947130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.967545] LNet: 69782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.977057] LNetError: 69782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.984367] LNetError: 69782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1368.995241] LNet: Removed LNI 192.168.206.9@tcp [ 1369.011984] LNet: Removed LNI 192.168.206.9@tcp1 [ 1369.050725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.120303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.161826] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1369.795833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1369.811188] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1370.560461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1370.610527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1370.636355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1370.672469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1370.709692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1370.748741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1370.804276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.9@tcp --health 0 [ 1370.900815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1371.168678] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1371.168773] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.9@tcp [ 1371.181464] LNet: Skipped 1 previous similar message [ 1386.263243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1386.426450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1386.484436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.515191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.9@tcp --health 500 [ 1386.624562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1393.273817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1393.321584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.341369] LNet: 70273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.347388] LNetError: 70273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.352786] LNetError: 70273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1393.358153] LNet: Removed LNI 192.168.206.9@tcp [ 1393.373932] LNet: Removed LNI 192.168.206.9@tcp1 [ 1393.393471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.451522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.479219] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1394.049680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1394.077088] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1394.844313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1394.938741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1395.000451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1395.033487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1395.068298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1395.101743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1395.136710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1395.175969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.9@tcp --health 0 [ 1396.513399] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1396.517951] LNet: Skipped 3 previous similar messages [ 1397.671492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1407.581295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1407.613488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1407.642027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1413.882806] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:11:07 (1787573467) [ 1414.287414] LNet: 71331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.301821] LNetError: 71331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.309062] LNetError: 71331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1414.320813] LNet: Removed LNI 192.168.206.9@tcp [ 1414.333926] LNet: Removed LNI 192.168.206.9@tcp1 [ 1415.053134] Key type .llcrypt unregistered [ 1415.056223] Key type ._llcrypt unregistered [ 1420.974578] Key type ._llcrypt registered [ 1420.976643] Key type .llcrypt registered [ 1421.092298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.102127] alg: No test for adler32 (adler32-zlib) [ 1422.004140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1422.037671] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1430.656861] Lustre: DEBUG MARKER: Initial discovery [ 1430.706968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.109@tcp [ 1432.981494] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1433.013853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1433.066923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.109@tcp [ 1433.089614] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1433.113612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1433.153689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.109@tcp -r 1 -m GET -l 3 [ 1433.180091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.109@tcp -r 1 -m GET -e local_error [ 1433.212943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.109@tcp -r 1 -m PUT -l 6 [ 1433.314475] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1434.220135] Lustre: DEBUG MARKER: Force oleg609-client.virtnet to discover 192.168.206.109@tcp (in background) [ 1434.533718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.109@tcp [ 1435.578960] Lustre: DEBUG MARKER: Force oleg609-server to discover 192.168.206.9@tcp [ 1437.331178] Lustre: DEBUG MARKER: Wait for 72100 [ 1437.599391] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1438.799712] Lustre: DEBUG MARKER: Finished wait on 72100 [ 1438.835129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1438.865590] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1438.869502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1439.075811] LNet: 72552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.080849] LNetError: 72552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.112191] LNet: Removed LNI 192.168.206.9@tcp [ 1439.640075] Key type .llcrypt unregistered [ 1439.645584] Key type ._llcrypt unregistered [ 1457.426075] Key type ._llcrypt registered [ 1457.428725] Key type .llcrypt registered [ 1458.660856] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:11:52 (1787573512) [ 1459.337346] Key type .llcrypt unregistered [ 1459.342331] Key type ._llcrypt unregistered [ 1465.143465] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1465.238209] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1465.257165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1465.265320] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1467.824485] Key type ._llcrypt registered [ 1467.828641] Key type .llcrypt registered [ 1468.010841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1468.023324] alg: No test for adler32 (adler32-zlib) [ 1468.937462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.993096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.047766] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1469.637122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1469.654964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1476.467609] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:12:10 (1787573530) [ 1476.578756] LNet: 74629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.583647] LNetError: 74629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1476.589096] LNet: Removed LNI 10.1.2.3@tcp [ 1476.595838] LNet: Removed LNI 192.168.206.9@tcp [ 1477.153496] Key type .llcrypt unregistered [ 1477.156291] Key type ._llcrypt unregistered [ 1483.568806] Key type ._llcrypt registered [ 1483.572735] Key type .llcrypt registered [ 1483.736751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1483.750426] alg: No test for adler32 (adler32-zlib) [ 1484.685079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.732832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.765956] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1485.439504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1485.461387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1486.621571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.9@tcp [ 1487.711161] LNetError: Unexpected error -22 connecting to 192.168.206.9@tcp at host 192.168.206.9:988 [ 1493.907227] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:12:27 (1787573547) [ 1494.216423] LNet: 75772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.222551] LNetError: 75772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1494.229430] LNet: Removed LNI 10.1.2.3@tcp [ 1494.237708] LNet: Removed LNI 192.168.206.9@tcp [ 1495.287437] Key type .llcrypt unregistered [ 1495.292201] Key type ._llcrypt unregistered [ 1503.354922] Key type ._llcrypt registered [ 1503.358699] Key type .llcrypt registered [ 1503.749546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1503.783887] alg: No test for adler32 (adler32-zlib) [ 1504.777127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.857225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1504.908967] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1506.094657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1506.120550] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1507.099477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.9@tcp1 --nid 192.168.206.9@tcp2 [ 1510.179980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp1 192.168.206.9@tcp2 [ 1510.202820] LNetError: 76612:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.9@tcp1 to 192.168.206.9@tcp2 - Route table may be misconfigured [ 1510.222259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp2 192.168.206.9@tcp1 [ 1515.835663] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:12:49 (1787573569) [ 1516.169195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.189166] LNet: 77186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.197300] LNetError: 77186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.201049] LNet: Removed LNI 192.168.206.9@tcp1 [ 1517.229523] LNet: Removed LNI 192.168.206.9@tcp2 [ 1517.290554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.406499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.481858] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1518.361481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.382324] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1519.195805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1519.215582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.9@tcp [ 1519.235676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp1 192.168.206.9@tcp1 [ 1519.275137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.9@tcp -d 192.168.206.9@tcp -e network_timeout [ 1519.307445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -e network_timeout [ 1519.356538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -e network_timeout [ 1519.394491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -e network_timeout [ 1519.425397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1519.438259] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1519.455402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1519.554778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.648102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1520.415565] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1524.762555] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:12:58 (1787573578) [ 1525.056291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.073597] LNet: 78047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.078529] LNetError: 78047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.084876] LNetError: 78047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1525.090983] LNet: Removed LNI 192.168.206.9@tcp [ 1526.127245] LNet: Removed LNI 192.168.206.9@tcp1 [ 1526.149031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.214668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1526.433628] LNet: 78088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.657619] Key type .llcrypt unregistered [ 1526.660273] Key type ._llcrypt unregistered [ 1535.440609] Key type ._llcrypt registered [ 1535.443846] Key type .llcrypt registered [ 1537.153351] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:13:10 (1787573590) [ 1537.860155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1537.886612] alg: No test for adler32 (adler32-zlib) [ 1538.792124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.960802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1542.354134] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:13:15 (1787573595) [ 1542.491904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.545777] LNet: 79180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.597857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.717388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.772679] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1543.895581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.935414] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1545.345602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1545.421962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp1 [ 1545.469409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp1 [ 1552.555798] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:13:26 (1787573606) [ 1554.724942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg609-server [ 1557.000553] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:13:30 (1787573610) [ 1558.778652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg609-server [ 1560.127025] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:13:34 (1787573614) [ 1561.503708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg609-server [ 1563.078889] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:13:37 (1787573617) [ 1565.038451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg609-server [ 1566.603716] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:13:40 (1787573620) [ 1568.156194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg609-server [ 1570.073111] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:13:43 (1787573623) [ 1572.401558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg609-server [ 1574.543192] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:13:48 (1787573628) [ 1576.649804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg609-server [ 1578.449274] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:13:52 (1787573632) [ 1580.728315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg609-server [ 1583.197668] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:13:56 (1787573636) [ 1585.194870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg609-server [ 1586.570333] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:14:00 (1787573640) [ 1586.805541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.829758] LNet: 83063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.846925] LNetError: 83063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.855417] LNet: Removed LNI 192.168.206.9@tcp [ 1587.879142] LNet: Removed LNI 192.168.206.9@tcp1 [ 1587.897956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.968915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.000085] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1588.553850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1588.619653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1588.764542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.780479] LNet: 83211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.794224] LNetError: 83211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.802218] LNetError: 83211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1588.823349] LNet: Removed LNI 192.168.206.9@tcp [ 1588.852462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.910357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1588.939408] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1589.558349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1589.805623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.824810] LNetError: 83356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.839468] LNet: Removed LNI 192.168.206.9@tcp [ 1589.857913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.896313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.922871] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1590.482730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1590.570818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1590.754204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.785939] LNet: Removed LNI 192.168.206.9@tcp [ 1591.832789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.899256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1591.945729] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1592.617841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1592.823266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.861829] LNet: 83648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.876894] LNet: 83648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1592.888945] LNetError: 83648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.899300] LNetError: 83648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1592.921907] LNet: Removed LNI 192.168.206.9@tcp [ 1592.948895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.045868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.065509] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1593.590694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1593.632238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1593.834494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.917764] LNet: Removed LNI 192.168.206.9@tcp [ 1594.949764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.999325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1595.023744] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1595.690161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1595.954915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.029960] LNet: Removed LNI 192.168.206.9@tcp [ 1597.062457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.118647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.165303] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1597.980440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1598.059144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1598.305409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.335108] LNet: 84088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.343848] LNet: 84088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1598.356838] LNetError: 84088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.370822] LNetError: 84088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1598.396642] LNet: Removed LNI 192.168.206.9@tcp [ 1598.415859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.457212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1598.474790] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1599.028525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1599.265676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.363329] LNet: Removed LNI 192.168.206.9@tcp [ 1600.394911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.434397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.480992] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1601.109946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1601.220446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1601.402207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.479381] LNet: Removed LNI 192.168.206.9@tcp [ 1602.514760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.601842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1602.628740] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1603.429840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1603.718896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.776443] LNet: Removed LNI 192.168.206.9@tcp [ 1604.803030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.858836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.884485] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1605.475541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1605.564605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1605.789947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.822763] LNet: Removed LNI 192.168.206.9@tcp [ 1606.862104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.934740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1606.982774] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1607.629397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1607.887737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.910797] LNet: 84819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.915917] LNet: 84819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1607.923880] LNetError: 84819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.931025] LNetError: 84819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1608.999819] LNet: Removed LNI 192.168.206.9@tcp [ 1609.029672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.083595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.113754] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1609.710294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1609.764450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.812687] LNet: Removed LNI 192.168.206.9@tcp [ 1609.829267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.873119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.906178] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1610.583893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1610.646384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.670605] LNet: Removed LNI 192.168.206.9@tcp [ 1610.684788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.725982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1610.804961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.849101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.903917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1610.982115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.022662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.096043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1611.184977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.215671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.252213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1611.280963] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1611.894507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.931125] LNet: Removed LNI 192.168.206.9@tcp [ 1611.955710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.016392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1618.007503] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:14:32 (1787573672) [ 1618.139839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.211386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.282706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1618.302885] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1619.354680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.406171] LNet: Removed LNI 192.168.206.9@tcp231 [ 1619.433279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.505659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1619.535465] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1619.542225] LNet: 85986:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1620.049094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1620.087286] LNet: Removed LNI 192.168.206.9@tcp231 [ 1620.116206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1620.144993] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1621.052735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.081762] LNet: Removed LNI 192.168.206.9@tcp231 [ 1621.097893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.137551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1621.155956] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/0] [ 1621.158933] LNet: 86138:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1621.556780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1621.587412] LNet: Removed LNI 192.168.206.9@tcp231 [ 1621.599901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1621.629768] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/0] [ 1622.574941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.613442] LNet: Removed LNI 192.168.206.9@tcp231 [ 1622.632429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.694247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1622.714414] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/60] [ 1622.723644] LNet: 86289:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1623.095127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1623.133756] LNet: Removed LNI 192.168.206.9@tcp231 [ 1623.149387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1623.167552] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/60] [ 1624.360988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.386313] LNet: 86433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.392692] LNet: 86433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1624.398873] LNetError: 86433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.403931] LNetError: 86433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1624.419963] LNet: Removed LNI 192.168.206.9@tcp231 [ 1624.439586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.493888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1624.522830] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1624.937871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1624.983860] LNet: Removed LNI 192.168.206.9@tcp231 [ 1625.013853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1625.031662] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1631.058445] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:14:45 (1787573685) [ 1631.175191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.248575] LNet: Removed LNI 192.168.206.9@tcp231 [ 1631.288974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.357601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1631.382540] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1631.932699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1632.002500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1632.035478] LNet: Removed LNI 192.168.206.9@tcp232 [ 1632.048337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1632.086620] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1632.699916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1632.739929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1632.772076] LNet: Removed LNI 192.168.206.9@tcp232 [ 1632.805573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1632.844915] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1633.535482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1633.593555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1633.621775] LNet: Removed LNI 192.168.206.9@tcp232 [ 1633.635642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1633.661583] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1634.273185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1634.332568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1634.374802] LNet: Removed LNI 192.168.206.9@tcp232 [ 1634.400086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1639.650140] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:14:53 (1787573693) [ 1640.883338] Key type .llcrypt unregistered [ 1640.893384] Key type ._llcrypt unregistered [ 1646.881325] Key type ._llcrypt registered [ 1646.883640] Key type .llcrypt registered [ 1647.063345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1647.075105] alg: No test for adler32 (adler32-zlib) [ 1647.916368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1647.937598] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1655.382989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1666.813284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.876148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1666.911854] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1669.140674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_interrupt -r 2 [ 1669.218992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 2 [ 1669.339856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_interrupt -r 2 [ 1669.396702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 2 [ 1669.457649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_interrupt -r 2 [ 1669.529893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 2 [ 1669.583314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_interrupt -r 2 [ 1669.678233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 2 [ 1669.771780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1669.818046] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1669.830634] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1670.241757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.345108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1670.405310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.463821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1670.506877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1670.585877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1670.663394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1670.755542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1670.801065] LNet: 88946:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1670.878317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1671.014058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1671.090777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1671.126280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1671.185824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1671.223676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1672.227898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_dropped -r 2 [ 1672.289493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 2 [ 1672.357320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_dropped -r 2 [ 1672.433536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 2 [ 1672.508879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_dropped -r 2 [ 1672.547288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 2 [ 1672.628622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_dropped -r 2 [ 1672.709030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 2 [ 1672.780969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1672.851793] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1673.287452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1673.339565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1673.386315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1673.456957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1673.516996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1673.568503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1673.602905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1673.650267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1673.671817] LNet: 89066:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1673.705974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1673.765675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1673.840493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1673.931693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1673.997621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1674.090608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1675.059803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_aborted -r 2 [ 1675.170952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 2 [ 1675.264357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_aborted -r 2 [ 1675.351674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 2 [ 1675.464000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_aborted -r 2 [ 1675.512746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 2 [ 1675.549862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_aborted -r 2 [ 1675.586491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 2 [ 1675.671970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1675.721550] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1676.105429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1676.136859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1676.181520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1676.221775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1676.266875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1676.316480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1676.361369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1676.432654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1676.472831] LNet: 89187:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1676.508017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1676.552803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1676.591895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1676.647996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1676.713132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1676.785868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1677.548280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_no_route -r 2 [ 1677.639058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 2 [ 1677.721421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_no_route -r 2 [ 1677.800215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 2 [ 1677.851719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_no_route -r 2 [ 1677.962367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 2 [ 1678.061169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_no_route -r 2 [ 1678.097497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 2 [ 1678.173832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1678.232461] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1678.505410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1678.561535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1678.604879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1678.657282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1678.713205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1678.750174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1678.799139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1678.833540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1678.878620] LNet: 89308:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1678.893357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1678.933281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1678.965593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1678.986461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1679.010585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1679.038837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1679.499280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_timeout -r 2 [ 1679.533470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 2 [ 1679.574027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_timeout -r 2 [ 1679.637718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 2 [ 1679.750332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_timeout -r 2 [ 1679.809892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 2 [ 1679.851940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_timeout -r 2 [ 1679.888316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 2 [ 1679.962851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1680.372926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1680.419867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1680.485471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1680.523732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1680.573554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1680.611448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1680.652509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1680.685619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1680.725226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1680.762449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1680.804196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1680.841300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1680.872799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1680.915593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1681.637485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_dropped -r 2 [ 1681.679399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 2 [ 1681.718801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_dropped -r 2 [ 1681.750653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 2 [ 1681.787572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_dropped -r 2 [ 1681.823793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 2 [ 1681.853538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_dropped -r 2 [ 1681.879557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 2 [ 1681.910710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1681.929621] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1682.120279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.169557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1682.218740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1682.264552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1682.307456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1682.355710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1682.392351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.423745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1682.454419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1682.505921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1682.541225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1682.579731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1682.608450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1682.649863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1687.599718] LNet: 89626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.604726] LNetError: 89626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.614707] LNet: Removed LNI 192.168.206.9@tcp [ 1687.624800] LNet: Removed LNI 192.168.206.9@tcp1 [ 1687.985624] Key type .llcrypt unregistered [ 1687.989720] Key type ._llcrypt unregistered [ 1701.526740] Key type ._llcrypt registered [ 1701.529508] Key type .llcrypt registered [ 1702.469060] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:15:56 (1787573756) [ 1702.843275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1702.851437] alg: No test for adler32 (adler32-zlib) [ 1703.758851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.800557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.828134] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1704.535481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1704.577050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.9@tcp [ 1704.590689] LNetError: 90606:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1704.666249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.9@tcp [ 1704.743793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1710.007973] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:16:04 (1787573764) [ 1710.189490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.209522] LNet: 91195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.217945] LNetError: 91195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.229881] LNet: Removed LNI 192.168.206.9@tcp [ 1710.244426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.288342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.313949] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1710.885740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1710.916900] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1711.686753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.9@tcp [ 1711.708402] LNetError: 91461:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1712.351453] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1712.356714] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1712.912806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.9@tcp [ 1720.292663] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:16:14 (1787573774) [ 1721.979958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg609-server [ 1723.862853] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:16:17 (1787573777) [ 1725.603953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg609-server [ 1727.748118] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:16:21 (1787573781) [ 1729.801364] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg609-server [ 1731.638144] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:16:25 (1787573785) [ 1731.721362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.759094] LNet: 93085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.764185] LNetError: 93085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.831322] LNet: Removed LNI 192.168.206.9@tcp [ 1732.836904] LNetError: 93085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.860225] LNet: Removed LNI 192.168.206.9@tcp1 [ 1732.919306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.061305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.094922] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1734.213539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1734.359588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1734.434880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1734.601956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1734.705826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1734.820631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1742.300113] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:16:35 (1787573795) [ 1742.542619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.613905] LNet: 93823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.617496] LNetError: 93823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.640732] LNet: Removed LNI 192.168.206.9@tcp [ 1742.709929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.802651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.867174] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1752.336384] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:16:46 (1787573806) [ 1752.487018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.514307] LNet: 94522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.525644] LNetError: 94522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.557355] LNet: Removed LNI 192.168.206.9@tcp [ 1752.586739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.661210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.695312] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1761.476278] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:16:54 (1787573814) [ 1763.126452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1765.443811] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:16:59 (1787573819) [ 1766.013764] LNet: 95564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.025727] LNetError: 95564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.050334] LNet: Removed LNI 192.168.206.9@tcp [ 1766.681605] Key type .llcrypt unregistered [ 1766.684749] Key type ._llcrypt unregistered [ 1772.906329] Key type ._llcrypt registered [ 1772.908163] Key type .llcrypt registered [ 1773.079295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1773.095316] alg: No test for adler32 (adler32-zlib) [ 1774.021587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1774.071104] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1781.251030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1785.724700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.109@tcp [ 1786.077124] LNet: 95990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.086765] LNetError: 95990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.108884] LNet: Removed LNI 192.168.206.9@tcp [ 1786.464347] Key type .llcrypt unregistered [ 1786.469182] Key type ._llcrypt unregistered [ 1797.003509] Key type ._llcrypt registered [ 1797.006306] Key type .llcrypt registered [ 1798.245480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1799.779679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1801.226188] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:17:35 (1787573855) [ 1802.845827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1804.381544] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:17:38 (1787573858) [ 1805.804789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg609-server [ 1807.328853] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:17:41 (1787573861) [ 1808.748843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg609-server [ 1810.180651] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:17:44 (1787573864) [ 1811.462184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.573376] alg: No test for adler32 (adler32-zlib) [ 1812.639710] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1812.830752] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1814.479200] Key type lgssc registered [ 1815.564600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.683644] Key type lgssc unregistered [ 1833.942990] LNet: 99167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.963481] LNetError: 99167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.975275] LNet: Removed LNI 192.168.206.9@tcp [ 1834.567464] Key type .llcrypt unregistered [ 1834.571965] Key type ._llcrypt unregistered [ 1848.669944] Key type ._llcrypt registered [ 1848.672430] Key type .llcrypt registered [ 1849.742164] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:18:23 (1787573903) [ 1850.043259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.056691] alg: No test for adler32 (adler32-zlib) [ 1850.936620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.993866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1851.029304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1851.071195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1851.116365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1856.310123] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:18:30 (1787573910) [ 1856.511287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.530494] LNet: 100582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.547831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.587979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.621682] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1857.367799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.392913] LNet: 100726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.400040] LNetError: 100726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.424712] LNet: Removed LNI 192.168.206.9@tcp [ 1857.456653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.506390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1857.524519] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1858.058803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.081236] LNetError: 100755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.099041] LNet: Removed LNI 192.168.206.9@tcp [ 1858.116755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.194343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1858.229097] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1864.098244] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:18:38 (1787573918) [ 1864.242387] LNet: 101345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.245991] LNet: 101345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1864.250843] LNetError: 101345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.269519] LNet: Removed LNI 192.168.206.9@tcp [ 1864.837707] Key type .llcrypt unregistered [ 1864.840495] Key type ._llcrypt unregistered [ 1871.790732] Key type ._llcrypt registered [ 1871.792100] Key type .llcrypt registered [ 1871.921840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1871.934897] alg: No test for adler32 (adler32-zlib) [ 1872.824794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.893310] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1872.899731] LNet: *** cfs_fail_loc=e003, val=100*** [ 1873.632685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1873.649297] LNet: *** cfs_fail_loc=e003, val=100*** [ 1873.651427] LNet: Skipped 2 previous similar messages [ 1873.661264] Ping reply received with mapping: [ 1873.662757] found metadata reply NI[192.168.206.9@tcp] [ 1873.665900] 09 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1873.668089] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1873.670524] 5a 5a 5a 5a 5a 5a 5a 5a [ 1873.749173] LNet: 101746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.754898] LNetError: 101746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.789680] LNet: Removed LNI 192.168.206.9@tcp [ 1875.405476] Key type .llcrypt unregistered [ 1875.407659] Key type ._llcrypt unregistered [ 1887.042872] Key type ._llcrypt registered [ 1887.045114] Key type .llcrypt registered [ 1888.060641] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:19:02 (1787573942) [ 1888.619129] Key type .llcrypt unregistered [ 1888.621957] Key type ._llcrypt unregistered [ 1895.382546] Key type ._llcrypt registered [ 1895.385201] Key type .llcrypt registered [ 1895.523345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1895.535157] alg: No test for adler32 (adler32-zlib) [ 1896.855954] LNetError: 102805:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1897.257226] Key type .llcrypt unregistered [ 1897.260091] Key type ._llcrypt unregistered [ 1902.356147] Key type ._llcrypt registered [ 1902.357908] Key type .llcrypt registered [ 1904.187732] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:19:17 (1787573957) [ 1904.934326] Key type .llcrypt unregistered [ 1904.945858] Key type ._llcrypt unregistered [ 1911.685608] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1911.703849] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1914.261676] Key type ._llcrypt registered [ 1914.266438] Key type .llcrypt registered [ 1914.443817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1914.458581] alg: No test for adler32 (adler32-zlib) [ 1915.349256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.399494] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1922.939790] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:19:37 (1787573977) [ 1923.188393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.224449] LNet: 104596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.238530] LNetError: 104596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1923.269679] LNet: Removed LNI 10.1.2.3@tcp [ 1923.300673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.937521] Key type .llcrypt unregistered [ 1923.946729] Key type ._llcrypt unregistered [ 1930.020687] Key type ._llcrypt registered [ 1930.022267] Key type .llcrypt registered [ 1930.214094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1930.237884] alg: No test for adler32 (adler32-zlib) [ 1931.295721] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1931.391844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1931.636884] LNet: 104889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.647883] LNetError: 104889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.668772] LNet: Removed LNI 192.168.206.9@tcp [ 1932.526588] Key type .llcrypt unregistered [ 1932.539724] Key type ._llcrypt unregistered [ 1947.169443] Key type ._llcrypt registered [ 1947.173728] Key type .llcrypt registered [ 1948.621588] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:20:02 (1787574002) [ 1949.376690] Key type .llcrypt unregistered [ 1949.380459] Key type ._llcrypt unregistered [ 1955.587099] Key type ._llcrypt registered [ 1955.589856] Key type .llcrypt registered [ 1955.926606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1955.943852] alg: No test for adler32 (adler32-zlib) [ 1963.008838] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:20:16 (1787574016) [ 1963.317810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.360913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.427392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.446665] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1964.007833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.046696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1964.079813] LNet: Removed LNI 192.168.206.9@tcp [ 1964.276205] LNet: 106646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.585673] Key type .llcrypt unregistered [ 1964.586926] Key type ._llcrypt unregistered [ 1975.382778] Key type ._llcrypt registered [ 1975.386650] Key type .llcrypt registered [ 1977.153117] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:20:30 (1787574030) [ 1977.589078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.600988] alg: No test for adler32 (adler32-zlib) [ 1978.528462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.645788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1978.687310] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1979.417960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1979.459372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.9@tcp [ 1979.490810] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1979.505941] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1979.508904] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1979, 0, 0 [ 1979.514921] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1979.517647] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1979.521405] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1979.523591] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1979, 0, 0 [ 1979.526761] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1979.528068] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1979.530689] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1979.532638] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1979, 0, 2029 [ 1979.535604] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1979.538185] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1979.541057] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1979.543509] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1979, 0, 2029 [ 1979.547088] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1986.294713] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:20:39 (1787574039) [ 1986.792513] LNet: 108186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.810307] LNetError: 108186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.828309] LNet: Removed LNI 192.168.206.9@tcp [ 1987.487159] Key type .llcrypt unregistered [ 1987.492156] Key type ._llcrypt unregistered [ 1993.935644] Key type ._llcrypt registered [ 1993.938046] Key type .llcrypt registered [ 1994.063678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1994.078500] alg: No test for adler32 (adler32-zlib) [ 1995.041155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1995.070425] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2002.110897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 2009.811897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.861820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2009.888427] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 2010.743711] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2010.938935] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2010.948905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2010.955633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2013.203819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2013.238617] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2023.177278] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:21:17 (1787574077) [ 2023.463739] LNet: 109700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.469504] LNetError: 109700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.481273] LNet: Removed LNI 192.168.206.9@tcp [ 2024.543242] LNet: Removed LNI 192.168.206.9@tcp1 [ 2024.547348] LNetError: 109700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2024.557078] LNetError: 109700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2024.571788] LNet: Removed LNI 10.1.2.3@tcp99 [ 2025.243725] Key type .llcrypt unregistered [ 2025.246599] Key type ._llcrypt unregistered [ 2031.152423] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2031.157803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2032.509330] Key type ._llcrypt registered [ 2032.511095] Key type .llcrypt registered [ 2032.623262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2032.638778] alg: No test for adler32 (adler32-zlib) [ 2033.565751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.637125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2033.681691] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2034.226378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2034.237796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2036.226469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2036.297535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2036.364278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2036.379600] LNetError: 110440:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2036.408473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2036.453190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2036.890982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2036.944816] LNet: 110460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.953736] LNetError: 110460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2036.959703] LNet: Removed LNI 10.1.2.3@tcp [ 2037.990349] LNet: Removed LNI 192.168.206.9@tcp [ 2038.845418] Key type .llcrypt unregistered [ 2038.848499] Key type ._llcrypt unregistered [ 2051.450826] Key type ._llcrypt registered [ 2051.460204] Key type .llcrypt registered [ 2052.759660] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:21:46 (1787574106) [ 2053.310540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.325542] alg: No test for adler32 (adler32-zlib) [ 2054.219237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.270592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.302721] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2054.948498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2055.081262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg609-client@tcp [ 2061.197814] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:21:55 (1787574115) [ 2061.427564] LNet: 111998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.439273] LNetError: 111998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.454547] LNet: Removed LNI 192.168.206.9@tcp [ 2062.122500] Key type .llcrypt unregistered [ 2062.125165] Key type ._llcrypt unregistered [ 2069.001863] Key type ._llcrypt registered [ 2069.004803] Key type .llcrypt registered [ 2069.158966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.181481] alg: No test for adler32 (adler32-zlib) [ 2070.185894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2070.220838] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2070.284531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2077.989276] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:22:11 (1787574131) [ 2079.835129] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2081.463493] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:22:15 (1787574135) [ 2083.248148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2085.143214] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:22:18 (1787574138) [ 2085.423230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.461245] LNet: 113425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.475619] LNetError: 113425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.493083] LNet: Removed LNI 192.168.206.9@tcp [ 2085.539721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.639759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2085.692475] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2086.405169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2086.455814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2093.785407] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:22:27 (1787574147) [ 2094.079498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.113541] LNet: 114134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.121625] LNetError: 114134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.141388] LNet: Removed LNI 192.168.206.9@tcp [ 2094.179086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.275163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2094.323255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2095.192642] Key type .llcrypt unregistered [ 2095.197367] Key type ._llcrypt unregistered [ 2106.133716] Key type ._llcrypt registered [ 2106.135650] Key type .llcrypt registered [ 2107.379570] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:22:41 (1787574161) [ 2107.716316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.737265] alg: No test for adler32 (adler32-zlib) [ 2108.664898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.751489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.775590] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2109.382285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2109.472934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2115.007169] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:22:48 (1787574168) [ 2115.189423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.205778] LNet: 115700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.214369] LNetError: 115700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.225636] LNet: Removed LNI 192.168.206.9@tcp [ 2115.251632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.315973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2115.342652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2121.369582] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:22:55 (1787574175) [ 2121.417111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.468693] LNet: 116272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.496179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.580789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2121.907468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2121.953139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2121.998237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2122.450886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2122.484851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2122.589589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.602401] LNet: 116313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.621389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.671108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2122.944139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2122.964918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2123.011835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2123.577451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2123.611563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.327333] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:23:03 (1787574183) [ 2129.497773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.550888] LNet: 116913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.585638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.658171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.681676] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2130.345745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2130.378938] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2131.353507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2131.431828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2131.641772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.676576] LNetError: 117192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.685408] LNet: Removed LNI 192.168.206.9@tcp [ 2131.698538] LNet: Removed LNI 192.168.206.9@tcp2 [ 2131.712826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.752772] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2131.760352] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2131.765303] LNet: 117198:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.150934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2132.350179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.407148] LNet: Removed LNI 192.168.206.9@tcp [ 2133.413288] LNetError: 117235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.420220] LNetError: 117235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2133.436920] LNet: Removed LNI 192.168.206.9@tcp2 [ 2133.475803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.560357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.579515] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2134.259867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2134.281394] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2134.948696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2134.983792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2135.052756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2135.302721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.324892] LNet: 117520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.330831] LNet: 117520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2136.353780] LNet: Removed LNI 192.168.206.9@tcp [ 2136.359292] LNetError: 117520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.373325] LNetError: 117520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2136.385883] LNet: Removed LNI 192.168.206.9@tcp2 [ 2136.421602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.472857] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2136.488669] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2136.494160] LNet: 117526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2136.904942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2137.092319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.144611] LNet: Removed LNI 192.168.206.9@tcp [ 2138.213648] LNet: Removed LNI 192.168.206.9@tcp2 [ 2138.251068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.329669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.362714] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2139.106118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2139.140845] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2139.901873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2139.948271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2140.145460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.216795] LNet: Removed LNI 192.168.206.9@tcp [ 2141.230337] LNetError: 117843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.236207] LNetError: 117843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2141.256077] LNet: Removed LNI 192.168.206.9@tcp2 [ 2141.329279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.405982] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2141.417077] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2141.428560] LNet: 117849:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2141.851991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2142.076709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.135465] LNet: Removed LNI 192.168.206.9@tcp [ 2143.148896] LNet: Removed LNI 192.168.206.9@tcp2 [ 2143.188941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.269894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.295250] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2144.051993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2144.077903] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2145.093167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2145.132717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2145.181593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2145.407103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.433318] LNet: 118170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.437833] LNet: 118170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2146.463171] LNet: Removed LNI 192.168.206.9@tcp [ 2147.495100] LNet: Removed LNI 192.168.206.9@tcp2 [ 2147.525659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.635091] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2147.647749] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2147.652927] LNet: 118177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2148.167844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2148.354293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.372686] LNet: Removed LNI 192.168.206.9@tcp [ 2149.414386] LNet: Removed LNI 192.168.206.9@tcp2 [ 2149.446661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.516662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.542818] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2150.246498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.270361] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2151.137520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2151.194631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2151.287922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2151.663887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.683272] LNetError: 118510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.689407] LNetError: 118510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2151.697769] LNet: Removed LNI 192.168.206.9@tcp [ 2151.719459] LNet: Removed LNI 192.168.206.9@tcp2 [ 2151.739669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.791487] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2151.796956] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2151.800799] LNet: 118516:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2152.567476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2153.017961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.079181] LNet: Removed LNI 192.168.206.9@tcp [ 2154.088820] LNet: Removed LNI 192.168.206.9@tcp2 [ 2154.131836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.224720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.265190] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2154.983689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2155.028649] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2155.887407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2155.947134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2155.983396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2156.026597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2156.362568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.407157] LNet: Removed LNI 192.168.206.9@tcp [ 2158.441460] LNet: Removed LNI 192.168.206.9@tcp2 [ 2158.480655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.551835] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2158.558626] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2159.083493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2166.197614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2167.592958] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:23:41 (1787574221) [ 2167.729014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.760507] LNet: 119634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.763874] LNet: 119634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2167.771564] LNetError: 119634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.775333] LNetError: 119634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2167.781521] LNet: Removed LNI 192.168.206.9@tcp [ 2168.805648] LNet: Removed LNI 192.168.206.9@tcp2 [ 2168.888861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.144471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2169.168704] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2169.877733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2169.906657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2170.624960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.654969] LNet: Removed LNI 10.1.2.3@tcp [ 2170.671763] LNet: Removed LNI 192.168.206.9@tcp [ 2170.697377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.762440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2170.785215] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2170.791944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2171.271924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2171.298878] LNet: Removed LNI 192.168.206.9@tcp [ 2171.323681] LNet: Removed LNI 10.1.2.3@tcp [ 2178.233315] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:23:51 (1787574231) [ 2178.312630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.376587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.423571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.457162] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2179.224272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2179.276554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.324981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.348240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.375843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.400747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.435156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.475977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.517320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.540539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.610632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 2179.850870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2187.988399] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:24:01 (1787574241) [ 2188.279463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.354275] LNet: Removed LNI 192.168.206.9@tcp [ 2189.383990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.457420] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2189.664450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2189.673451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2189.681275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2192.081872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.106980] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2192.733970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2192.751181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2194.351439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2194.381569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2194.440794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.9@tcp -m PUT -e local_timeout -r 1 [ 2194.574783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2194.717363] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2194.721948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2195.096553] LNetError: 121313:0:(lib-move.c:4801:lnet_parse()) 192.168.206.9@tcp, src 192.168.206.9@tcp: Dropping PUT (error -108 looking up sender) [ 2196.128858] LNet: Removed LNI 10.1.2.3@tcp [ 2197.169776] LNet: Removed LNI 192.168.206.9@tcp [ 2197.559742] Key type .llcrypt unregistered [ 2197.563593] Key type ._llcrypt unregistered [ 2214.459652] Key type ._llcrypt registered [ 2214.463934] Key type .llcrypt registered [ 2216.482571] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:24:29 (1787574269) [ 2216.989145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.054525] alg: No test for adler32 (adler32-zlib) [ 2217.957936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.491486] LNet: 122631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.543447] Key type .llcrypt unregistered [ 2219.547939] Key type ._llcrypt unregistered [ 2226.800451] Key type ._llcrypt registered [ 2226.802861] Key type .llcrypt registered [ 2227.003155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2227.013990] alg: No test for adler32 (adler32-zlib) [ 2227.926551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2227.951612] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2237.171744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 2237.351620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2238.022523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.109@tcp [ 2238.175170] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 2238.192689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2238.240387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2238.287571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2238.337576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2238.377922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2238.410549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2238.444853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2238.481321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2238.494819] LNet: 123085:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2238.506632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2238.558130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2238.606821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2238.636114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2238.660658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2238.710952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2239.213161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2239.392674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2239.421213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2239.469625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2239.500184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2239.537113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2239.562955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2239.599103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2239.637739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2239.656765] LNet: 123202:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.667058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2239.696949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2239.737597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2239.759870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2239.789498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2239.811298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2240.265610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.109@tcp [ 2240.457334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2240.484490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2240.516821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2240.543522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2240.572535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2240.603823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2240.648897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2240.691079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2240.701814] LNet: 123319:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2240.713037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2240.730775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2240.754571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2240.797586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2240.823665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2240.851487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2241.262669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2241.519340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2241.549720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2241.575163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2241.597466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2241.624919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2241.666750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2241.697132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2241.734560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2241.767533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2241.802481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2241.851337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2241.881511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2241.904432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2241.934199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2242.287948] LNet: 123486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.297801] LNetError: 123486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.310785] LNet: Removed LNI 192.168.206.9@tcp [ 2242.641553] Key type .llcrypt unregistered [ 2242.644527] Key type ._llcrypt unregistered [ 2258.506850] Key type ._llcrypt registered [ 2258.508440] Key type .llcrypt registered [ 2259.750650] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:25:13 (1787574313) [ 2260.126762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2260.141819] alg: No test for adler32 (adler32-zlib) [ 2261.062562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.427745] LNet: 124373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.948902] Key type .llcrypt unregistered [ 2261.950512] Key type ._llcrypt unregistered [ 2267.601918] Key type ._llcrypt registered [ 2267.604691] Key type .llcrypt registered [ 2267.763679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2267.777735] alg: No test for adler32 (adler32-zlib) [ 2268.674756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2268.695236] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2275.883855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 2284.582242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.672316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2284.712938] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 2285.684786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2286.280685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.109@tcp [ 2286.624693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.668502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2286.705439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.762924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.819839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.852798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.888632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.929945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.948172] LNet: 125029:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.978834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2287.019278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2287.060424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2287.113204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2287.136340] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 2287.154304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2287.208225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2287.993287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2288.226944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.267714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.318603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.361299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.396483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.429833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.473666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.532841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.562150] LNet: 125146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.585203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.626037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.671510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.739390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.822969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.861356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2289.618486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.109@tcp [ 2289.939042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.976747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2290.016353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2290.043294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2290.090336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2290.138380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2290.175892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2290.223888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2290.259760] LNet: 125263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2290.281643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2290.316541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2290.358322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2290.392820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2290.426986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2290.465353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2291.161561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2291.405282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.437348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2291.471650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.522642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.570754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.601513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.639878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.668844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.711657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.751664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.794978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.835289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.900385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2291.968560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2297.231695] LNet: 125466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.238951] LNetError: 125466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.248817] LNet: Removed LNI 192.168.206.9@tcp [ 2297.268108] LNet: Removed LNI 192.168.206.9@tcp1 [ 2297.721815] Key type .llcrypt unregistered [ 2297.724312] Key type ._llcrypt unregistered [ 2313.826612] Key type ._llcrypt registered [ 2313.827984] Key type .llcrypt registered [ 2314.873384] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:26:08 (1787574368) [ 2315.282984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.294380] alg: No test for adler32 (adler32-zlib) [ 2316.219673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.271048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2316.295369] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2316.912286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2316.928360] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 2317.789811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 2317.845579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2318.879399] LNet: Removed LNI 192.168.206.9@tcp [ 2324.903746] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:26:18 (1787574378) [ 2326.323539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2327.735085] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:26:21 (1787574381) [ 2334.730869] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:26:28 (1787574388) [ 2341.250757] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:26:35 (1787574395) [ 2347.407909] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:26:41 (1787574401) [ 2353.060849] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:26:47 (1787574407) [ 2358.591812] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:26:52 (1787574412) [ 2364.729838] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:26:58 (1787574418) [ 2370.265469] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:27:04 (1787574424) [ 2376.126358] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:27:10 (1787574430) [ 2381.576853] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:27:15 (1787574435) [ 2386.536149] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:27:20 (1787574440) [ 2393.186442] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:27:27 (1787574447) [ 2398.997307] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:27:32 (1787574452) [ 2404.413657] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:27:38 (1787574458) [ 2410.248609] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:27:44 (1787574464) [ 2416.071355] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:27:50 (1787574470) [ 2424.079303] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:27:57 (1787574477) [ 2433.475051] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:28:06 (1787574486) [ 2433.650601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.687591] LNet: 137153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.703039] LNetError: 137153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.732739] LNet: Removed LNI 192.168.206.9@tcp2 [ 2433.795995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.919670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2433.943888] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2435.448663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2435.477637] LNet: 137299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.497350] LNetError: 137299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.512608] LNet: Removed LNI 192.168.206.9@tcp [ 2435.543767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.613816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2435.647404] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2442.351964] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:28:16 (1787574496) [ 2449.554839] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:28:23 (1787574503) [ 2457.677700] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:28:31 (1787574511) [ 2465.188666] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:28:38 (1787574518) [ 2472.091050] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:28:45 (1787574525) [ 2479.482741] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:28:53 (1787574533) [ 2487.030454] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:29:01 (1787574541) [ 2493.907867] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:29:07 (1787574547) [ 2501.032614] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:29:14 (1787574554) [ 2508.395455] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:29:22 (1787574562) [ 2515.144562] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:29:28 (1787574568) [ 2522.339220] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:29:35 (1787574575) [ 2530.184654] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2262 sec ========== 08:29:43 (1787574583) [ 2530.642211] LNet: 144765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.655368] LNetError: 144765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.677052] LNet: Removed LNI 192.168.206.9@tcp [ 2531.550665] Key type .llcrypt unregistered [ 2531.560124] Key type ._llcrypt unregistered [ 2543.749742] Key type ._llcrypt registered [ 2543.756089] Key type .llcrypt registered [ 2544.279709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2544.290205] alg: No test for adler32 (adler32-zlib) [ 2545.375512] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2545.661280] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2547.376716] Key type lgssc registered [ 2548.624900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2615.811333] Lustre: Mounted lustre-client - version 2.17.57_80_g9b8a413 [ 2620.248517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2627.539932] Lustre: Unmounted lustre-client [ 2656.552702] Key type lgssc unregistered [ 2656.946079] LNet: 147733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.954968] LNetError: 147733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.980194] LNet: Removed LNI 192.168.206.9@tcp [ 2657.935163] Key type .llcrypt unregistered [ 2657.938683] Key type ._llcrypt unregistered