[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 436098790 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003162] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008213] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011165] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.015038] *** VALIDATE selinux *** [ 0.024063] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029641] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031020] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032143] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033152] *** VALIDATE tmpfs *** [ 0.034592] *** VALIDATE proc *** [ 0.035307] *** VALIDATE cgroup *** [ 0.036018] *** VALIDATE cgroup2 *** [ 0.037346] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040044] Spectre V2 : User space: Vulnerable [ 0.041015] Speculative Store Bypass: Vulnerable [ 0.044826] debug: unmapping init [mem 0xffffffffb3659000-0xffffffffb3660fff] [ 0.047237] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048815] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049035] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051023] ... generic registers: 4 [ 0.052022] ... value mask: 0000ffffffffffff [ 0.053023] ... max period: 00007fffffffffff [ 0.054023] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.057283] rcu: Hierarchical SRCU implementation. [ 0.059727] smp: Bringing up secondary CPUs ... [ 0.060713] x86: Booting SMP configuration: [ 0.061043] .... node #0, CPUs: #1 #2 #3 [ 0.065387] smp: Brought up 1 node, 4 CPUs [ 0.067022] smpboot: Max logical packages: 1 [ 0.068021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230078] node 0 deferred pages initialised in 159ms [ 0.233117] devtmpfs: initialized [ 0.234317] x86/mm: Memory block size: 128MB [ 0.238096] gcov: version magic: 0x41383552 [ 0.240175] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246314] pinctrl core: initialized pinctrl subsystem [ 0.248188] [ 0.248608] ************************************************************* [ 0.251015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253013] ** ** [ 0.255016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258015] ** ** [ 0.260014] ** This means that this kernel is built to expose internal ** [ 0.262013] ** IOMMU data structures, which may compromise security on ** [ 0.264016] ** your system. ** [ 0.267014] ** ** [ 0.269011] ** If you see this message and you are not debugging the ** [ 0.271014] ** kernel, report this immediately to your vendor! ** [ 0.274016] ** ** [ 0.276012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278015] ************************************************************* [ 0.280596] NET: Registered protocol family 16 [ 0.282422] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.285062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.288061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.291449] cpuidle: using governor menu [ 0.292425] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294570] PCI: Using configuration type 1 for base access [ 0.297125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.312167] cryptd: max_cpu_qlen set to 1000 [ 0.316043] ACPI: Added _OSI(Module Device) [ 0.317011] ACPI: Added _OSI(Processor Device) [ 0.319014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.321010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.325168] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.331100] ACPI: Interpreter enabled [ 0.332053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333012] ACPI: Using IOAPIC for interrupt routing [ 0.335091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.337353] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347406] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.350042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.352018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.361296] acpiphp: Slot [2] registered [ 0.363124] acpiphp: Slot [5] registered [ 0.364151] acpiphp: Slot [6] registered [ 0.366145] acpiphp: Slot [3] registered [ 0.368103] acpiphp: Slot [4] registered [ 0.369125] acpiphp: Slot [7] registered [ 0.370121] acpiphp: Slot [8] registered [ 0.372104] acpiphp: Slot [9] registered [ 0.373100] acpiphp: Slot [10] registered [ 0.375128] acpiphp: Slot [11] registered [ 0.376099] acpiphp: Slot [12] registered [ 0.378126] acpiphp: Slot [13] registered [ 0.379124] acpiphp: Slot [14] registered [ 0.380143] acpiphp: Slot [15] registered [ 0.382102] acpiphp: Slot [16] registered [ 0.383092] acpiphp: Slot [17] registered [ 0.385098] acpiphp: Slot [18] registered [ 0.386097] acpiphp: Slot [19] registered [ 0.388099] acpiphp: Slot [20] registered [ 0.389119] acpiphp: Slot [21] registered [ 0.391115] acpiphp: Slot [22] registered [ 0.392102] acpiphp: Slot [23] registered [ 0.395138] acpiphp: Slot [24] registered [ 0.396098] acpiphp: Slot [25] registered [ 0.398119] acpiphp: Slot [26] registered [ 0.400116] acpiphp: Slot [27] registered [ 0.401248] acpiphp: Slot [28] registered [ 0.403115] acpiphp: Slot [29] registered [ 0.404101] acpiphp: Slot [30] registered [ 0.405127] acpiphp: Slot [31] registered [ 0.407064] PCI host bridge to bus 0000:00 [ 0.409022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.416028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.419024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.422023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.424026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.426172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.433242] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.440016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.445073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.447020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.449019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.452022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.454537] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.458734] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.461038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.463820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.467938] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.476025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.481016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485891] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.493015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.498017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.520421] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.527015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.546018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.560328] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.564419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.567277] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.569421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.571220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.575165] iommu: Default domain type: Passthrough [ 0.577442] SCSI subsystem initialized [ 0.579148] ACPI: bus type USB registered [ 0.580090] usbcore: registered new interface driver usbfs [ 0.583105] usbcore: registered new interface driver hub [ 0.585096] usbcore: registered new device driver usb [ 0.586160] pps_core: LinuxPPS API ver. 1 registered [ 0.588014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.590055] PTP clock support registered [ 0.592207] EDAC MC: Ver: 3.0.0 [ 0.595021] PCI: Using ACPI for IRQ routing [ 0.596867] NetLabel: Initializing [ 0.597018] NetLabel: domain hash size = 128 [ 0.598016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.599159] NetLabel: unlabeled traffic allowed by default [ 0.601108] vgaarb: loaded [ 0.602379] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.604424] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.612306] clocksource: Switched to clocksource kvm-clock [ 0.722712] VFS: Disk quotas dquot_6.6.0 [ 0.724453] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.727231] *** VALIDATE ramfs *** [ 0.728855] *** VALIDATE hugetlbfs *** [ 0.730943] pnp: PnP ACPI init [ 0.733835] pnp: PnP ACPI: found 6 devices [ 0.753141] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.757269] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.759545] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.762223] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.764923] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.767502] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.770616] NET: Registered protocol family 2 [ 0.773351] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.779163] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.783217] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.788927] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.792469] TCP: Hash tables configured (established 65536 bind 65536) [ 0.795892] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.799227] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.802126] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.805074] NET: Registered protocol family 1 [ 0.807818] RPC: Registered named UNIX socket transport module. [ 0.809705] RPC: Registered udp transport module. [ 0.811574] RPC: Registered tcp transport module. [ 0.813285] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.816095] NET: Registered protocol family 44 [ 0.817849] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.819888] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.822075] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.824511] PCI: CLS 0 bytes, default 64 [ 0.826300] Unpacking initramfs... [ 2.299539] debug: unmapping init [mem 0xffff93a9fcc64000-0xffff93a9fffcffff] [ 2.303210] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.305072] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.308186] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.812300] Initialise system trusted keyrings [ 2.814462] Key type blacklist registered [ 2.816809] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.827534] zbud: loaded [ 2.831100] *** VALIDATE nfs *** [ 2.832158] *** VALIDATE nfs4 *** [ 2.834381] pstore: using deflate compression [ 2.838976] Platform Keyring initialized [ 2.947593] NET: Registered protocol family 38 [ 2.949237] Key type asymmetric registered [ 2.950985] Asymmetric key parser 'x509' registered [ 2.952816] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.956168] io scheduler mq-deadline registered [ 2.958263] io scheduler kyber registered [ 2.960246] io scheduler bfq registered [ 2.962478] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.965450] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.968324] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.971416] ACPI: Power Button [PWRF] [ 2.976735] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.984984] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.001483] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.031288] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.059259] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.065038] Non-volatile memory driver v1.3 [ 3.066797] Linux agpgart interface v0.103 [ 3.108973] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.112661] vda: detected capacity change from 0 to 74760192 [ 3.127993] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.131087] vdb: detected capacity change from 0 to 1073741824 [ 3.138630] libphy: Fixed MDIO Bus: probed [ 3.144368] usbcore: registered new interface driver usbserial_generic [ 3.147040] usbserial: USB Serial support registered for generic [ 3.149137] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.152805] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.154659] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.157291] mousedev: PS/2 mouse device common for all mice [ 3.159809] rtc_cmos 00:05: RTC can wake from S4 [ 3.161664] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.163314] rtc_cmos 00:05: registered as rtc0 [ 3.167715] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.170671] intel_pstate: CPU model not supported [ 3.173524] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.175198] hid: raw HID events driver (C) Jiri Kosina [ 3.178901] usbcore: registered new interface driver usbhid [ 3.179052] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.181042] usbhid: USB HID core driver [ 3.181249] drop_monitor: Initializing network drop monitor service [ 3.187965] Initializing XFRM netlink socket [ 3.190175] NET: Registered protocol family 10 [ 3.193060] Segment Routing with IPv6 [ 3.194686] NET: Registered protocol family 17 [ 3.196901] mpls_gso: MPLS GSO support [ 3.202850] RAS: Correctable Errors collector initialized. [ 3.205346] AVX version of gcm_enc/dec engaged. [ 3.206876] AES CTR mode by8 optimization enabled [ 3.285990] sched_clock: Marking stable (3285966395, 0)->(4205944784, -919978389) [ 3.289640] registered taskstats version 1 [ 3.291935] Loading compiled-in X.509 certificates [ 3.294378] zswap: loaded using pool lzo/zbud [ 3.319431] Key type big_key registered [ 3.332453] Key type encrypted registered [ 3.333734] ima: No TPM chip found, activating TPM-bypass! [ 3.335849] ima: Allocated hash algorithm: sha1 [ 3.337472] ima: No architecture policies found [ 3.339227] evm: Initialising EVM extended attributes: [ 3.341267] evm: security.selinux [ 3.342496] evm: security.ima [ 3.343605] evm: security.capability [ 3.344889] evm: HMAC attrs: 0x1 [ 3.347094] rtc_cmos 00:05: setting system clock to 2026-08-31 12:06:48 UTC (1788178008) [ 3.353145] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.356235] debug: unmapping init [mem 0xffffffffb3382000-0xffffffffb3658fff] [ 3.369268] Write protecting the kernel read-only data: 28672k [ 3.372904] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 3.375651] debug: unmapping init [mem 0xffffffffb2314000-0xffffffffb23fffff] [ 3.408613] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.415847] systemd[1]: Detected virtualization kvm. [ 3.417372] systemd[1]: Detected architecture x86-64. [ 3.418972] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.445661] systemd[1]: No hostname configured. [ 3.447592] systemd[1]: Set hostname to . [ 3.450235] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452572] systemd[1]: Initializing machine ID from random generator. [ 3.605170] random: systemd: uninitialized urandom read (16 bytes read) [ 3.607580] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.612477] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614413] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.618619] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.275566] device-mapper: uevent: version 1.0.3 [ 4.278255] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 4.840455] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.046780] virtio_net virtio0 ens2: renamed from eth0 [ 5.120769] scsi host0: ata_piix [ 5.126696] scsi host1: ata_piix [ 5.128272] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.130219] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.364405] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.855694] random: crng init done [ 9.857178] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.226314] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.309641] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.571743] SELinux: Disabled at runtime. [ 11.631047] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.640396] systemd[1]: Detected virtualization kvm. [ 11.641823] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.134330] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.137666] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.143255] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.147297] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.150073] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.159532] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.168650] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK [0[ 12.320160] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.554090] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.852563] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.897615] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.976330] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.989697] EDAC sbridge: Ver: 1.1.2 [ 14.164975] Key type dns_resolver registered [ 14.473120] NFS: Registering the id_resolver key type [ 14.475207] Key type id_resolver registered [ 14.476624] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg652-client login: [ 45.562430] libcfs: loading out-of-tree module taints kernel. [ 45.717718] Key type ._llcrypt registered [ 45.723272] Key type .llcrypt registered [ 46.653969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 46.673595] alg: No test for adler32 (adler32-zlib) [ 48.413368] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 49.751218] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 51.711234] Key type lgssc registered [ 53.925947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 180.897789] Lustre: Mounted lustre-client [ 187.089785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 200.976371] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing check_logdir /tmp/testlogs/ [ 206.319276] Lustre: lustre-OST0000-osc-ffff93aa42c01000: disconnect after 23s idle [ 207.007613] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing yml_node [ 210.221234] hrtimer: interrupt took 1312627 ns [ 211.738337] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 214.393322] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 216.962897] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 218.792849] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 31 08:10:22 EDT 2026 [ 234.302409] Lustre: Unmounted lustre-client [ 252.998222] Key type lgssc unregistered [ 253.262941] LNet: 8375:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 253.279813] LNetError: 8375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 253.328589] LNet: Removed LNI 192.168.206.52@tcp [ 254.248216] Key type .llcrypt unregistered [ 254.250890] Key type ._llcrypt unregistered [ 277.724994] Key type ._llcrypt registered [ 277.728735] Key type .llcrypt registered [ 277.887866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 277.916206] alg: No test for adler32 (adler32-zlib) [ 278.901804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 278.986176] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 279.028943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 283.482459] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 283.541710] LNet: 9253:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.558169] LNetError: 9253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 283.581979] LNet: Removed LNI 192.168.206.52@tcp [ 284.139631] Key type .llcrypt unregistered [ 284.144166] Key type ._llcrypt unregistered [ 290.587707] Key type ._llcrypt registered [ 290.595025] Key type .llcrypt registered [ 291.824102] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:11:35 (1788178295) [ 292.216318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 292.228736] alg: No test for adler32 (adler32-zlib) [ 293.106046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.175822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.203758] LNet: 9670:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.229725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.311895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 293.335925] LNet: 9677:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 300.263986] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:11:44 (1788178304) [ 300.385995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.423833] LNet: 10247:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.467350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.605223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 308.259726] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:11:51 (1788178311) [ 308.338489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.352782] LNet: 10833:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.381883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.491846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 314.979604] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:11:58 (1788178318) [ 315.101684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.148293] LNet: 11419:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.182824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.280364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 323.142577] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:12:06 (1788178326) [ 323.228363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.282352] LNet: 12001:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.329330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.451298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 323.996610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 331.086792] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:12:14 (1788178334) [ 331.191969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.248807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.368065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 331.963843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 332.852244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 339.871069] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:12:23 (1788178343) [ 340.151953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.178807] LNet: 13197:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.186837] LNet: 13197:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 340.217636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.318498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 346.370229] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:12:30 (1788178350) [ 346.580932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.644707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.714235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 346.749306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 347.256550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 347.296942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 347.803944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 347.871192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 348.379474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 348.439498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 348.789528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 348.826162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 349.443793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 349.480425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 349.953101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 349.997487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 350.531723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 350.577124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 351.016945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 351.052211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 351.291576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 351.337482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 351.728805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 351.766670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 352.297526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 352.337928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 352.796619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 352.831712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 359.460089] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:12:43 (1788178363) [ 359.532460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.554576] LNet: 14456:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.570369] LNet: 14456:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 359.593760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.671975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 359.712813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 365.462573] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:12:49 (1788178369) [ 365.558922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.603618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.718556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 365.757232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 372.570501] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:12:56 (1788178376) [ 372.658690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.716834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.843967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 372.897257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 379.360503] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:13:03 (1788178383) [ 379.591703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.620846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.733803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 379.773731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 387.163930] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:13:10 (1788178390) [ 387.295746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.370675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.495616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 387.536580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 394.561093] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:13:18 (1788178398) [ 394.789824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.809323] LNet: 17394:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.813517] LNet: 17394:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 394.827868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.891538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 394.955642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 402.808311] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:13:26 (1788178406) [ 402.963557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.020474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.139459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.662393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 410.866434] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:13:34 (1788178414) [ 411.286632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.327807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.526609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 411.780155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 418.914111] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:13:42 (1788178422) [ 419.339290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.401991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.541273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 419.825756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.616900] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:13:51 (1788178431) [ 427.865228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.934385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.484526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.741994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.786417] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:13:59 (1788178439) [ 436.938695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.995671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.198208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.303749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.187502] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:14:08 (1788178448) [ 445.357968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.465945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.633304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.756109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.681893] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:14:19 (1788178459) [ 455.860796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.944445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.086076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 465.203534] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:14:28 (1788178468) [ 465.276612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.316382] LNet: 22247:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.323595] LNet: 22247:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 465.350839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.430503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 472.758584] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:14:36 (1788178476) [ 472.860686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.941970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.118219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 473.739760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 480.908764] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:14:44 (1788178484) [ 481.105702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.177242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.294737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 481.334145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 488.974554] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:14:52 (1788178492) [ 489.110343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.195449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.329274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 489.378348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 496.335558] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:15:00 (1788178500) [ 496.446232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.494479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.577712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 496.652615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 504.107559] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:15:07 (1788178507) [ 504.321551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.389135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.586801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 504.645575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 512.861525] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:15:16 (1788178516) [ 513.026264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.104479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.159515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 513.222936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 513.242387] LNetError: 25786:0:(peer.c:2516:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 513.282249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 513.318853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 519.477854] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:15:23 (1788178523) [ 519.579021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.650183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.762733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.trk9LBowpc [ 519.855496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.trk9LBowpc [ 519.906351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.trk9LBowpc [ 526.580139] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:15:30 (1788178530) [ 526.787376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.818948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.857525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 526.885883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 526.921801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 533.324605] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:15:36 (1788178536) [ 533.539827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.624370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.755219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 533.793530] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 535.290462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 536.902877] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 538.823164] Key type lgssc registered [ 540.345660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 544.379360] LNetError: 27548:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.206.52@tcp99 has no available nets [ 547.065331] Key type lgssc unregistered [ 547.525607] LNetError: 28656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.549420] LNet: Removed LNI 192.168.206.52@tcp [ 548.227748] Key type .llcrypt unregistered [ 548.231966] Key type ._llcrypt unregistered [ 555.172672] Key type ._llcrypt registered [ 555.174373] Key type .llcrypt registered [ 556.781735] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:16:00 (1788178560) [ 557.060187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 557.077603] alg: No test for adler32 (adler32-zlib) [ 558.075811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.232479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 561.913092] LNet: Message forwarding will be enabled in 2s [ 565.024108] LNet: Message forwarding enabled [ 565.453746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 565.502767] LNet: Message forwarding will stop in 2s to 6s [ 570.335696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 570.925022] LNet: Message forwarding disabled [ 573.337656] LNet: Message forwarding will be enabled in 2s [ 577.058320] LNet: Message forwarding enabled [ 580.629737] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:16:23 (1788178583) [ 580.720447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.753749] LNet: 30059:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.348179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.480568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 589.915420] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:16:33 (1788178593) [ 590.028360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.077484] LNet: 30639:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.117853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.196997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 590.255418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 590.294912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 590.330686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 590.364158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 590.398624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 590.440282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 590.496334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 590.541706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 590.609986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 590.662446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 590.692892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 590.735309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 590.770339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 590.824606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 590.856995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 590.899461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 590.956288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 591.010955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 591.039402] LNetError: 30691:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 597.839197] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:16:41 (1788178601) [ 597.933455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.969946] LNet: 31255:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.000184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.142550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.365885] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:16:49 (1788178609) [ 607.411551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg652-server [ 609.257360] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:16:53 (1788178613) [ 609.475641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.496707] LNet: 32193:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.515429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.559098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.578330] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 610.430637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[53-57]@tcp [ 610.450201] LNet: 32326:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 610.465314] LNetError: 32204:0:(lib-lnet.h:1291:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.53@tcp has gone from up to down [ 610.518485] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.54@tcp is being used as a gateway but routing feature is not turned on [ 611.508456] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.56@tcp is being used as a gateway but routing feature is not turned on [ 611.520619] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 612.519300] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.56@tcp is being used as a gateway but routing feature is not turned on [ 612.536677] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 614.567097] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 614.591526] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 617.239717] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:17:00 (1788178620) [ 618.669796] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.54@tcp is being used as a gateway but routing feature is not turned on [ 618.699135] LNetError: 32197:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 619.159140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg652-server [ 621.116166] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:17:04 (1788178624) [ 621.297259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.326921] LNet: 33262:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.341811] LNetError: 33262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.362554] LNet: Removed LNI 192.168.206.52@tcp [ 621.398429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.483819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.545913] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 622.256974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[53-58/2]@tcp [ 622.274118] LNet: 33379:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 622.280243] LNet: 33379:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 622.287170] LNetError: 33270:0:(lib-lnet.h:1291:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.53@tcp has gone from up to down [ 622.300932] LNetError: 33270:0:(lib-lnet.h:1291:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 622.317927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[53-58/2]@tcp [ 628.967402] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:17:12 (1788178632) [ 629.165912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.208686] LNetError: 33972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.231481] LNet: Removed LNI 192.168.206.52@tcp [ 629.261889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.420426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 629.534495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.565205] LNet: 33986:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.575988] LNet: 33986:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 629.612978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.699879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.818263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.879379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.947905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 630.091465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.177632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.321793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.426804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.506765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.577248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 630.713453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.788559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.863698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.011311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.049787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.142349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 631.253152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.297607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.382442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.489418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.552803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.667959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.796858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.856722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.977960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 632.073839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.113586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.182991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 632.234078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.309589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.419918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.628560] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:17:22 (1788178642) [ 641.856856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg652-server [ 643.812787] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:17:27 (1788178647) [ 646.018409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg652-server [ 648.120924] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:17:31 (1788178651) [ 648.212533] LNet: 35364:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.217325] LNet: 35364:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 648.831687] Key type .llcrypt unregistered [ 648.833996] Key type ._llcrypt unregistered [ 656.082845] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 656.227344] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 656.232936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 656.238952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 658.965564] Key type ._llcrypt registered [ 658.970028] Key type .llcrypt registered [ 659.183683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 659.205635] alg: No test for adler32 (adler32-zlib) [ 660.140495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.190319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 660.231694] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 661.215710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 661.253051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 663.256339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 663.287221] LNet: Removed LNI 10.1.2.3@tcp [ 663.401291] LNet: 36096:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.408737] LNetError: 36096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.423449] LNet: Removed LNI 192.168.206.52@tcp [ 664.156734] Key type .llcrypt unregistered [ 664.160727] Key type ._llcrypt unregistered [ 676.573444] Key type ._llcrypt registered [ 676.576159] Key type .llcrypt registered [ 677.980928] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:18:01 (1788178681) [ 678.469575] Key type .llcrypt unregistered [ 678.475865] Key type ._llcrypt unregistered [ 685.277430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 685.554893] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 685.561281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 685.568811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 688.196424] Key type ._llcrypt registered [ 688.197987] Key type .llcrypt registered [ 688.388213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 688.402858] alg: No test for adler32 (adler32-zlib) [ 689.363397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.426825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.464048] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 690.307388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 690.342976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 693.043523] LNet: 37591:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.048894] LNetError: 37591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 693.055214] LNet: Removed LNI 10.1.2.3@tcp [ 693.068650] LNet: Removed LNI 192.168.206.52@tcp [ 693.872508] Key type .llcrypt unregistered [ 693.878900] Key type ._llcrypt unregistered [ 705.964500] Key type ._llcrypt registered [ 705.968087] Key type .llcrypt registered [ 707.477587] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:18:30 (1788178710) [ 708.086968] Key type .llcrypt unregistered [ 708.093291] Key type ._llcrypt unregistered [ 714.622647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 714.989889] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 715.009772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 715.024488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 717.915386] Key type ._llcrypt registered [ 717.918188] Key type .llcrypt registered [ 718.208208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 718.244630] alg: No test for adler32 (adler32-zlib) [ 719.208795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.294371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 719.344412] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 722.239985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.267704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.312978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.328446] LNet: Removed LNI 10.1.2.3@tcp [ 725.343923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 725.380252] LNet: Removed LNI 10.1.2.31@tcp [ 725.416796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 725.454922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.633572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 726.700301] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 727.959647] LNet: 39531:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.963919] LNetError: 39531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 727.969194] LNet: Removed LNI 10.1.2.31@tcp [ 727.994957] LNet: Removed LNI 10.1.2.3@tcp [ 728.676572] Key type .llcrypt unregistered [ 728.678288] Key type ._llcrypt unregistered [ 741.568389] Key type ._llcrypt registered [ 741.571619] Key type .llcrypt registered [ 743.269984] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:19:06 (1788178746) [ 743.877738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 743.920348] alg: No test for adler32 (adler32-zlib) [ 744.984601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.095763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.140409] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 746.564799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.615279] LNet: 40485:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.639652] LNetError: 40485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.672966] LNet: Removed LNI 192.168.206.52@tcp [ 746.720883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.814446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.52@tcp [ 746.861922] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 748.228405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.52@tcp [ 748.272818] LNet: Removed LNI 192.168.206.52@tcp [ 748.396833] LNet: 40620:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.412069] Key type .llcrypt unregistered [ 749.418371] Key type ._llcrypt unregistered [ 762.748554] Key type ._llcrypt registered [ 762.751161] Key type .llcrypt registered [ 764.532295] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:19:27 (1788178767) [ 766.549969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg652-server [ 768.651421] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:19:32 (1788178772) [ 769.182427] Key type .llcrypt unregistered [ 769.187558] Key type ._llcrypt unregistered [ 775.790382] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 775.992097] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.007812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.016436] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.609281] Key type ._llcrypt registered [ 778.619764] Key type .llcrypt registered [ 778.825504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.849916] alg: No test for adler32 (adler32-zlib) [ 779.785585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.835097] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 780.779683] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 783.314273] LNet: 42452:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.325487] LNetError: 42452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.334184] LNet: Removed LNI 192.168.206.52@tcp1 [ 783.342167] LNet: Removed LNI 10.1.2.3@tcp2 [ 783.953376] Key type .llcrypt unregistered [ 783.955893] Key type ._llcrypt unregistered [ 796.411848] Key type ._llcrypt registered [ 796.416127] Key type .llcrypt registered [ 797.748343] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:20:01 (1788178801) [ 798.155268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.182347] alg: No test for adler32 (adler32-zlib) [ 799.063171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.113922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.146472] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 800.004613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.022267] LNet: 43423:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.027221] LNetError: 43423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.047702] LNet: Removed LNI 192.168.206.52@tcp [ 800.061392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.118862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 800.135959] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 807.222150] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:20:10 (1788178810) [ 807.364801] LNet: 44007:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.371780] LNetError: 44007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.417251] LNet: Removed LNI 192.168.206.52@tcp [ 808.086482] Key type .llcrypt unregistered [ 808.088954] Key type ._llcrypt unregistered [ 813.732764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.970357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.974536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.980096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.458427] Key type ._llcrypt registered [ 816.460054] Key type .llcrypt registered [ 816.664269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.677185] alg: No test for adler32 (adler32-zlib) [ 817.622363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.709786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.762099] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 818.542102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 818.574867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.725736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.742761] LNet: 44756:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.748786] LNetError: 44756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 820.752453] LNet: Removed LNI 10.1.2.3@tcp [ 820.761582] LNet: Removed LNI 192.168.206.52@tcp [ 820.791344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.884721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 820.908419] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 820.915548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.656779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.682150] LNet: 44783:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.696881] LNetError: 44783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 821.701925] LNetError: 44783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 821.706760] LNet: Removed LNI 10.1.2.3@tcp [ 821.714953] LNet: Removed LNI 192.168.206.52@tcp [ 821.750623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.801491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 821.821328] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 821.826416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.216578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 822.243059] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 823.157841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.186725] LNet: 44945:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.199051] LNetError: 44945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 823.209757] LNetError: 44945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 823.215120] LNet: Removed LNI 10.1.2.3@tcp [ 823.217193] LNet: Removed LNI 192.168.206.52@tcp [ 823.241691] LNet: Removed LNI 192.168.206.52@tcp2 [ 823.255653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.340646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 823.359823] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 823.366974] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.371505] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 830.812481] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:20:34 (1788178834) [ 830.987783] LNet: 45543:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.993225] LNetError: 45543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.997652] LNetError: 45543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 831.002446] LNet: Removed LNI 10.1.2.3@tcp [ 831.006488] LNet: Removed LNI 192.168.206.52@tcp [ 831.017725] LNet: Removed LNI 192.168.206.52@tcp2 [ 831.591927] Key type .llcrypt unregistered [ 831.595880] Key type ._llcrypt unregistered [ 838.611528] Key type ._llcrypt registered [ 838.613533] Key type .llcrypt registered [ 838.943391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.986378] alg: No test for adler32 (adler32-zlib) [ 839.977529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.029508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.051308] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 841.167976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.205339] LNet: 45937:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.218113] LNetError: 45937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.242317] LNet: Removed LNI 192.168.206.52@tcp [ 841.284297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.358964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 841.381160] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 841.872366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.902551] LNet: 45968:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.907680] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.938766] LNet: Removed LNI 192.168.206.52@tcp [ 841.963083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.036241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 842.085772] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 842.591817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.624674] LNet: Removed LNI 192.168.206.52@tcp [ 842.658940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.727183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 842.751956] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 843.281039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.462697] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.470430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.481523] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.872148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.893463] LNet: 46076:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.899735] LNet: 46076:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 845.908304] LNetError: 46076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.916817] LNetError: 46076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.930811] LNet: Removed LNI 192.168.206.52@tcp [ 845.961455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.044518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 846.081076] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 852.449355] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:20:56 (1788178856) [ 852.600814] LNet: 46668:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.609351] LNetError: 46668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.638318] LNet: Removed LNI 192.168.206.52@tcp [ 853.533899] Key type .llcrypt unregistered [ 853.537470] Key type ._llcrypt unregistered [ 859.731194] Key type ._llcrypt registered [ 859.737193] Key type .llcrypt registered [ 859.956729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.975631] alg: No test for adler32 (adler32-zlib) [ 860.948238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.004864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.028910] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 862.006579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.034405] LNet: 47071:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.047811] LNetError: 47071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.073537] LNet: Removed LNI 192.168.206.52@tcp [ 862.118513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.182706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 862.201976] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 869.455590] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:21:13 (1788178873) [ 869.691623] LNet: 47658:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.701067] LNetError: 47658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.730224] LNet: Removed LNI 192.168.206.52@tcp [ 870.792796] Key type .llcrypt unregistered [ 870.795374] Key type ._llcrypt unregistered [ 877.697633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.885417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.896052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.922307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.548579] Key type ._llcrypt registered [ 880.551564] Key type .llcrypt registered [ 880.756243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.773702] alg: No test for adler32 (adler32-zlib) [ 881.737069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.807987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.824722] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 882.682696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.702567] LNet: 48113:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.710876] LNetError: 48113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.727857] LNet: Removed LNI 192.168.206.52@tcp [ 882.748957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.811412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 882.829915] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 889.556281] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:21:33 (1788178893) [ 889.725683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.749818] LNet: 48709:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.763604] LNetError: 48709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.793518] LNet: Removed LNI 192.168.206.52@tcp [ 889.826976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.954637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 896.074533] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:21:39 (1788178899) [ 896.373882] LNet: 49293:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.907621] Key type .llcrypt unregistered [ 896.911832] Key type ._llcrypt unregistered [ 902.564961] Key type ._llcrypt registered [ 902.569790] Key type .llcrypt registered [ 902.827291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.853921] alg: No test for adler32 (adler32-zlib) [ 903.795979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.881807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.915224] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 904.949923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.986482] LNet: 49691:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.001525] LNetError: 49691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.022370] LNet: Removed LNI 192.168.206.52@tcp [ 905.046563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.140565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 905.202800] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 913.536673] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:21:56 (1788178916) [ 913.655898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.672929] LNet: 50279:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.694124] LNetError: 50279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.731900] LNet: Removed LNI 192.168.206.52@tcp [ 913.775141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.881401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 921.889159] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:22:05 (1788178925) [ 923.488864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 926.252374] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:22:09 (1788178929) [ 928.128492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 930.180323] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:22:13 (1788178933) [ 931.937603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 934.344873] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:22:17 (1788178937) [ 935.973511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 937.618325] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:22:21 (1788178941) [ 939.545148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 942.177425] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:22:25 (1788178945) [ 942.305240] LNet: 52332:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.235934] Key type .llcrypt unregistered [ 943.240098] Key type ._llcrypt unregistered [ 949.244397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.425871] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.434777] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.439696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.999747] Key type ._llcrypt registered [ 952.005571] Key type .llcrypt registered [ 952.202018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.211854] alg: No test for adler32 (adler32-zlib) [ 953.154981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.230926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.254344] LNet: Added LNI 192.168.206.52@tcp [8/512/0/180] [ 953.261461] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 953.267161] LNet: 52663:0:(api-ni.c:2978:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.374696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.398468] LNet: 52682:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.405422] LNetError: 52682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.412979] LNet: Removed LNI 10.1.2.3@tcp [ 953.415928] LNetError: 52682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.445646] LNet: Removed LNI 192.168.206.52@tcp [ 953.478462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.578814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.599732] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 953.672271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.698237] LNet: 52706:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.709130] LNetError: 52706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.733872] LNet: Removed LNI 192.168.206.52@tcp [ 953.756928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.810885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 953.851251] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 953.856651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.858140] LNet: 52714:0:(api-ni.c:2978:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 953.981062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.026423] LNet: 52732:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.036922] LNetError: 52732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.051228] LNet: Removed LNI 10.1.2.3@tcp [ 954.056650] LNetError: 52732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.092034] LNet: Removed LNI 192.168.206.52@tcp [ 954.127214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.195580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.212959] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 954.217224] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.219774] LNet: 52740:0:(api-ni.c:2978:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 954.436466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.473910] LNet: 52766:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.486307] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.502637] LNet: Removed LNI 10.1.2.3@tcp [ 954.504580] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.526710] LNet: Removed LNI 192.168.206.52@tcp [ 954.541701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.628386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.657210] LNet: Added LNI 192.168.206.52@tcp [8/256/0/182] [ 954.664170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 954.769708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.790933] LNet: 52794:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.797809] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.809501] LNet: Removed LNI 10.1.2.3@tcp [ 954.811475] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.824851] LNet: Removed LNI 192.168.206.52@tcp [ 954.839457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.905292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 954.926113] LNet: Added LNI 192.168.206.52@tcp [8/256/0/182] [ 954.929936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 954.934511] LNet: 52802:0:(api-ni.c:2978:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 955.006079] LNet: Removed LNI 10.1.2.3@tcp [ 955.021229] LNet: Removed LNI 192.168.206.52@tcp [ 955.693138] Key type .llcrypt unregistered [ 955.698046] Key type ._llcrypt unregistered [ 969.022559] Key type ._llcrypt registered [ 969.028843] Key type .llcrypt registered [ 970.510941] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:22:53 (1788178973) [ 970.887340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 970.904897] alg: No test for adler32 (adler32-zlib) [ 971.856880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.916595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 971.936929] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 972.864635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.52 [ 972.907881] LNet: 53778:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 972.918493] LNet: Removed LNI 192.168.206.52@tcp [ 979.314648] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:23:03 (1788178983) [ 979.473093] LNet: 54342:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.481379] LNetError: 54342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.519655] LNet: Removed LNI 192.168.206.52@tcp [ 980.353590] Key type .llcrypt unregistered [ 980.356290] Key type ._llcrypt unregistered [ 988.089144] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 988.385917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 991.234961] Key type ._llcrypt registered [ 991.237934] Key type .llcrypt registered [ 991.527929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.589251] alg: No test for adler32 (adler32-zlib) [ 993.379698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1000.891339] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:23:24 (1788179004) [ 1001.114763] LNet: 55262:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.137211] LNetError: 55262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1001.168477] LNet: Removed LNI 10.1.2.3@tcp [ 1001.983523] Key type .llcrypt unregistered [ 1001.991937] Key type ._llcrypt unregistered [ 1009.175492] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1009.424257] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1009.429954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1009.441620] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1012.196087] Key type ._llcrypt registered [ 1012.199181] Key type .llcrypt registered [ 1012.446681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.464644] alg: No test for adler32 (adler32-zlib) [ 1013.562688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1021.422471] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:23:45 (1788179025) [ 1023.328619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1025.139569] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:23:48 (1788179028) [ 1025.343267] LNet: 56450:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.356098] LNetError: 56450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.370909] LNet: Removed LNI 10.1.2.3@tcp [ 1026.170306] Key type .llcrypt unregistered [ 1026.172669] Key type ._llcrypt unregistered [ 1032.529237] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.682567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.686924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.697975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.501873] Key type ._llcrypt registered [ 1035.508715] Key type .llcrypt registered [ 1035.708488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.727747] alg: No test for adler32 (adler32-zlib) [ 1036.691912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.741423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1045.877964] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:24:09 (1788179049) [ 1046.318398] LNet: 57681:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.331652] LNetError: 57681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.346524] LNet: Removed LNI 10.1.2.3@tcp [ 1046.937675] Key type .llcrypt unregistered [ 1046.942173] Key type ._llcrypt unregistered [ 1053.148801] Key type ._llcrypt registered [ 1053.152760] Key type .llcrypt registered [ 1053.470841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.495296] alg: No test for adler32 (adler32-zlib) [ 1054.404207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1054.430793] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1060.967097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1061.511587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_interrupt -r 1 [ 1061.549881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_interrupt -r 1 [ 1061.577326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1061.597588] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1061.831056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.869845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.907991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.958521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.997982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.034702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.080394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.117747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.138497] LNet: 58119:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.159442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.193750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.238351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.279283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.311426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.364393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.405391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.991501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_dropped -r 1 [ 1063.050875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_dropped -r 1 [ 1063.099851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1063.118676] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1063.286912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.318461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.347553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.370643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.394837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.422760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.491937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.540701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.562424] LNet: 58228:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.581877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.610420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.632574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.647131] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1063.662271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.684697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.707343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.731314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.320528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_aborted -r 1 [ 1064.393593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_aborted -r 1 [ 1064.440369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1064.465070] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1064.698235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.738655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.791082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.832204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.874500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.900529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.928043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.984457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.015836] LNet: 58337:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.047516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.104672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.159924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.218923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.268767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.298206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.326208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1065.930200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_no_route -r 1 [ 1065.974636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_no_route -r 1 [ 1066.013491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1066.316458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.358580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.411870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1066.472468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1066.511275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1066.547585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.630871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.707878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.761075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.812931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.857326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.892635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.952805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.996896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.034811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1067.568840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_timeout -r 1 [ 1067.594660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_timeout -r 1 [ 1067.618878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1067.633482] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1067.639092] LNet: Skipped 2 previous similar messages [ 1067.808864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.827462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.856086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.885492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.908402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.937940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.959784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.981927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.994826] LNet: 58556:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.998664] LNet: 58556:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1068.014524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.044475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.078396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.112415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.160936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.216628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.246365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1068.696970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_error -r 1 [ 1068.728040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_error -r 1 [ 1068.776959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1068.986989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.013730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.053314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.080329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.104356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.126590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.151379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.182690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.203944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.225996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.256047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.297365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.348878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.379450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.404726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.819699] LNet: 58706:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.832121] LNetError: 58706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.862988] LNet: Removed LNI 192.168.206.52@tcp [ 1070.445642] Key type .llcrypt unregistered [ 1070.452951] Key type ._llcrypt unregistered [ 1085.781468] Key type ._llcrypt registered [ 1085.785164] Key type .llcrypt registered [ 1086.939835] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:24:50 (1788179090) [ 1087.998375] Key type .llcrypt unregistered [ 1088.005291] Key type ._llcrypt unregistered [ 1093.789728] Key type ._llcrypt registered [ 1093.791544] Key type .llcrypt registered [ 1094.011780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.034589] alg: No test for adler32 (adler32-zlib) [ 1095.016966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1095.042457] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1101.657889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1110.630648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1110.693398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1110.723569] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1112.219395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_interrupt -r 1 [ 1112.270356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_interrupt -r 1 [ 1112.320967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_interrupt -r 1 [ 1112.351417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_interrupt -r 1 [ 1112.382954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_interrupt -r 1 [ 1112.424370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_interrupt -r 1 [ 1112.467075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_interrupt -r 1 [ 1112.495077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_interrupt -r 1 [ 1112.531300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1112.557312] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1112.562540] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1112.855842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.899667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.940426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.975573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.013612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.063628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.103505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.154741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.184537] LNet: 60175:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.209984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.267814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.317296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.347220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.370498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.396170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.421190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.134540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_dropped -r 1 [ 1114.173263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_dropped -r 1 [ 1114.201134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_dropped -r 1 [ 1114.228566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_dropped -r 1 [ 1114.254606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_dropped -r 1 [ 1114.281409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_dropped -r 1 [ 1114.307805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_dropped -r 1 [ 1114.339442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_dropped -r 1 [ 1114.363345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1114.377762] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1114.386072] LNet: Skipped 2 previous similar messages [ 1114.619614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.662586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.702147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.730750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.780609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.814889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.849514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.879445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.899041] LNet: 60297:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.943050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.981939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.029932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.067497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.128877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.174293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.219510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.869172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_aborted -r 1 [ 1115.915823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_aborted -r 1 [ 1115.963684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_aborted -r 1 [ 1116.034549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_aborted -r 1 [ 1116.089308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_aborted -r 1 [ 1116.145889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_aborted -r 1 [ 1116.208583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_aborted -r 1 [ 1116.263199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_aborted -r 1 [ 1116.305888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1116.334429] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1116.339629] LNet: Skipped 2 previous similar messages [ 1116.601962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.633445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.672823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.699414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.733969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.757949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.783063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.811656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.826864] LNet: 60418:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.840970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.868668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.897405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.929808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.962953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.996359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.076317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.602933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_no_route -r 1 [ 1117.642305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_no_route -r 1 [ 1117.710277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_no_route -r 1 [ 1117.748904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_no_route -r 1 [ 1117.777935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_no_route -r 1 [ 1117.812922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_no_route -r 1 [ 1117.893315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_no_route -r 1 [ 1117.941471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_no_route -r 1 [ 1118.003990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1118.294992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.351112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.425479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.491474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.517910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.551525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.631788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.695648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.766816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.804162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.826495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.863230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.915798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.950600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.991371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.528348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_timeout -r 1 [ 1119.591327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_timeout -r 1 [ 1119.680824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_timeout -r 1 [ 1119.718927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_timeout -r 1 [ 1119.757271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_timeout -r 1 [ 1119.789070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_timeout -r 1 [ 1119.841810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_timeout -r 1 [ 1119.875516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_timeout -r 1 [ 1119.926249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1119.944148] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1119.954122] LNet: Skipped 5 previous similar messages [ 1120.167967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.220193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.257857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.281769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.301884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.331703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.369532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.405288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.426770] LNet: 60661:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.433431] LNet: 60661:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1120.453498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.477658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.519697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.551408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.601430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.632793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.662339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.216404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_error -r 1 [ 1121.262292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_error -r 1 [ 1121.293271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_error -r 1 [ 1121.343078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_error -r 1 [ 1121.399369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_error -r 1 [ 1121.452584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_error -r 1 [ 1121.486907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_error -r 1 [ 1121.534891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_error -r 1 [ 1121.606770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1121.897426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.928853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.966199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.015933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.044216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.091439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.118173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.143908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.167598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.195521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.219947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.240213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.269418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.308770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.331167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.639305] LNet: 60858:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.657181] LNetError: 60858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.675321] LNet: Removed LNI 192.168.206.52@tcp [ 1128.679197] LNetError: 60858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.712128] LNet: Removed LNI 192.168.206.52@tcp1 [ 1129.196066] Key type .llcrypt unregistered [ 1129.198988] Key type ._llcrypt unregistered [ 1144.767573] Key type ._llcrypt registered [ 1144.768967] Key type .llcrypt registered [ 1146.310100] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:25:49 (1788179149) [ 1147.617495] Key type .llcrypt unregistered [ 1147.625674] Key type ._llcrypt unregistered [ 1154.136838] Key type ._llcrypt registered [ 1154.138891] Key type .llcrypt registered [ 1154.396277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.435634] alg: No test for adler32 (adler32-zlib) [ 1155.395427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1155.436925] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1163.209800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1164.140076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_dropped -r 1 [ 1164.202573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_dropped -r 1 [ 1164.249150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1164.280122] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1164.516894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.552475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.587304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.609819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.639946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.682498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.717622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.742948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.757564] LNet: 62110:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1164.771047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.802850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.833951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.875058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.918752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.951068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.998345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1165.473403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_error -r 1 [ 1165.500259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_error -r 1 [ 1165.527631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1165.544933] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1165.827241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.858702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.916755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.951986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.987319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.018511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.055503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.106857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.133410] LNet: 62219:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.158914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.198537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.226147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.262983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.334821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.378570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.454651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.175092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_timeout -r 1 [ 1167.218989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_timeout -r 1 [ 1167.283252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1167.320897] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1167.556731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.601963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.671196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.723467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.787955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.842199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.900096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.955431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.006669] LNet: 62328:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.051810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.109753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.182565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.253554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.287060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.324415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.352566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1168.549487] LNet: 62369:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.556736] LNetError: 62369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.571347] LNet: Removed LNI 192.168.206.52@tcp [ 1168.905654] Key type .llcrypt unregistered [ 1168.909540] Key type ._llcrypt unregistered [ 1183.518404] Key type ._llcrypt registered [ 1183.520534] Key type .llcrypt registered [ 1184.859942] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:26:28 (1788179188) [ 1185.726208] Key type .llcrypt unregistered [ 1185.729386] Key type ._llcrypt unregistered [ 1191.836359] Key type ._llcrypt registered [ 1191.841182] Key type .llcrypt registered [ 1192.046759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.056045] alg: No test for adler32 (adler32-zlib) [ 1192.946349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.982188] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1199.368306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1208.747910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.829367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1208.861247] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1210.877290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_dropped -r 1 [ 1210.921795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e remote_dropped -r 1 [ 1210.957819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_dropped -r 1 [ 1211.029179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e remote_dropped -r 1 [ 1211.088471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e remote_dropped -r 1 [ 1211.166944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e remote_dropped -r 1 [ 1211.237416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e remote_dropped -r 1 [ 1211.304470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e remote_dropped -r 1 [ 1211.389337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1211.442817] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1211.452236] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.152@tcp [ 1211.702828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.735552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.757861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.779682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.826949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.877778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.935605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.992461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.018403] LNet: 63837:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.040865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.092457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.131146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.205206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.258913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.296983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.336327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.389350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.137312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_error -r 1 [ 1213.235388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e remote_error -r 1 [ 1213.297386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_error -r 1 [ 1213.358768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e remote_error -r 1 [ 1213.429665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e remote_error -r 1 [ 1213.529278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e remote_error -r 1 [ 1213.601874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e remote_error -r 1 [ 1213.668611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e remote_error -r 1 [ 1213.733428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1213.762601] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1213.769293] LNet: Skipped 2 previous similar messages [ 1214.066369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.119353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.181048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.251689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.279535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.306857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.364567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.404323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.428368] LNet: 63960:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.445827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.504034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.547292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.603635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.713395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.775878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.841882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.895880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.525503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_timeout -r 1 [ 1215.581368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e remote_timeout -r 1 [ 1215.636260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_timeout -r 1 [ 1215.694756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e remote_timeout -r 1 [ 1215.763870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e remote_timeout -r 1 [ 1215.809757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e remote_timeout -r 1 [ 1215.886898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e remote_timeout -r 1 [ 1215.965157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e remote_timeout -r 1 [ 1216.031896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.152@tcp [ 1216.064614] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1216.074019] LNet: Skipped 1 previous similar message [ 1216.374899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.439922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.483641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.537328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.568367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.603604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.633535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.662791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.678851] LNet: 64084:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.694447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.716920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.751952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.804823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.842171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.901711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.936705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1216.990749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.920641] LNet: 64162:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.929213] LNetError: 64162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.934030] LNet: Removed LNI 192.168.206.52@tcp [ 1221.950632] LNet: Removed LNI 192.168.206.52@tcp1 [ 1222.413641] Key type .llcrypt unregistered [ 1222.415993] Key type ._llcrypt unregistered [ 1237.882204] Key type ._llcrypt registered [ 1237.887115] Key type .llcrypt registered [ 1239.055668] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:27:22 (1788179242) [ 1240.590369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1242.055247] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:27:26 (1788179246) [ 1243.114629] Key type .llcrypt unregistered [ 1243.116191] Key type ._llcrypt unregistered [ 1248.897962] Key type ._llcrypt registered [ 1248.901079] Key type .llcrypt registered [ 1249.062737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.078385] alg: No test for adler32 (adler32-zlib) [ 1249.994924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1250.035183] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1257.681675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1258.604660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e network_timeout -r 1 [ 1258.672305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e network_timeout -r 1 [ 1258.728051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1258.777013] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1259.147547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.193453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.245118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.297139] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1259.302391] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1259.311731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.369930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.406162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.448505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.487198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.512538] LNet: 65705:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.535888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.573066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.619732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.664946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.697389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.723285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.000702] LNet: 65744:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.010244] LNetError: 65744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.036614] LNet: Removed LNI 192.168.206.52@tcp [ 1260.470940] Key type .llcrypt unregistered [ 1260.473571] Key type ._llcrypt unregistered [ 1278.731503] Key type ._llcrypt registered [ 1278.739414] Key type .llcrypt registered [ 1278.987102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.000972] alg: No test for adler32 (adler32-zlib) [ 1279.891757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1279.915613] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1287.428775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1296.952272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.998639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1297.033243] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1298.285465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e network_timeout -r 1 [ 1298.315768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e network_timeout -r 1 [ 1298.349397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e network_timeout -r 1 [ 1298.393969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e network_timeout -r 1 [ 1298.450861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e network_timeout -r 1 [ 1298.497452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e network_timeout -r 1 [ 1298.541733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e network_timeout -r 1 [ 1298.595672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e network_timeout -r 1 [ 1298.629749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1298.645809] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1298.828913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1298.857268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1298.882803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.905575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.955755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.991389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.018530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.047913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.082484] LNet: 66625:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.112216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.146284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.192965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.234726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.273815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.325796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1303.868483] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1304.283786] LNet: 66699:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.296876] LNetError: 66699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.304121] LNet: Removed LNI 192.168.206.52@tcp [ 1304.316874] LNet: Removed LNI 192.168.206.52@tcp1 [ 1304.834616] Key type .llcrypt unregistered [ 1304.840152] Key type ._llcrypt unregistered [ 1323.126579] Key type ._llcrypt registered [ 1323.128504] Key type .llcrypt registered [ 1324.359817] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:28:48 (1788179328) [ 1324.996510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.041757] alg: No test for adler32 (adler32-zlib) [ 1326.019825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.086232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1326.110392] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1326.943810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1326.966851] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1327.874928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 1327.960791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1328.041543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1328.115154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1328.178900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1328.213276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1328.269418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.52@tcp [ 1328.314481] LNetError: 67800:0:(api-ni.c:4475:lnet_ni_set_healthv()) manually adding local NI 192.168.206.52@tcp to recovery [ 1329.119279] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1329.120479] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1331.167512] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1331.974627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1335.263742] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1335.553738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1335.596875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1335.683508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.725248] LNet: 67907:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.737485] LNetError: 67907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.799461] LNet: Removed LNI 192.168.206.52@tcp [ 1336.803882] LNetError: 67907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.838500] LNet: Removed LNI 192.168.206.52@tcp1 [ 1336.855263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.946101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1336.989644] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1337.968618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.988493] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1339.130590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 1339.184840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.314573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1339.356580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1339.382388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1339.418233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1339.450836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1339.474778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.52@tcp [ 1339.498700] LNetError: 68180:0:(api-ni.c:4475:lnet_ni_set_healthv()) manually adding local NI 192.168.206.52@tcp to recovery [ 1339.999456] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1342.055932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1345.119407] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1345.125962] LNet: Skipped 1 previous similar message [ 1350.145972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1350.179296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1350.205928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.463319] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:29:20 (1788179360) [ 1357.570047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.614876] LNet: 68880:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.636843] LNetError: 68880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.642405] LNet: Removed LNI 192.168.206.52@tcp [ 1357.655373] LNet: Removed LNI 192.168.206.52@tcp1 [ 1357.704171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.782306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.823042] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1358.708275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.732528] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1359.687527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 1359.801570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1359.845736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1359.922870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1359.977699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1360.047365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1360.116263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.52@tcp --health 0 [ 1360.312691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1360.799305] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.52@tcp [ 1361.826772] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1361.840713] LNet: Skipped 1 previous similar message [ 1376.090600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1376.325921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.367517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.416359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.52@tcp --health 500 [ 1376.570601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.352678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.425466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.451600] LNet: 69372:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.455655] LNetError: 69372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.462770] LNetError: 69372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1382.470808] LNet: Removed LNI 192.168.206.52@tcp [ 1382.479594] LNet: Removed LNI 192.168.206.52@tcp1 [ 1382.505976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.588536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1382.623517] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1383.594504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.637934] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1384.752695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 1384.787357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1384.855805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1384.892465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1384.925513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1384.953338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1384.979895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.014214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.52@tcp --health 0 [ 1386.658092] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1386.670786] LNet: Skipped 3 previous similar messages [ 1387.472151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.005304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.056762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1398.100299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1404.826253] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:30:08 (1788179408) [ 1405.744903] LNet: 70430:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.769609] LNetError: 70430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.777885] LNetError: 70430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1405.784111] LNet: Removed LNI 192.168.206.52@tcp [ 1405.829374] LNet: Removed LNI 192.168.206.52@tcp1 [ 1406.902113] Key type .llcrypt unregistered [ 1406.903782] Key type ._llcrypt unregistered [ 1414.792984] Key type ._llcrypt registered [ 1414.794971] Key type .llcrypt registered [ 1415.144862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1415.179816] alg: No test for adler32 (adler32-zlib) [ 1416.196978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1416.268733] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1428.200841] Lustre: DEBUG MARKER: Initial discovery [ 1428.344219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.152@tcp [ 1431.164787] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1431.580912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1431.639616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.152@tcp [ 1431.686113] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1431.738764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1431.840380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.152@tcp -r 1 -m GET -l 3 [ 1431.944564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.152@tcp -r 1 -m GET -e local_error [ 1432.022677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.152@tcp -r 1 -m PUT -l 6 [ 1432.673982] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1433.596543] Lustre: DEBUG MARKER: Force oleg652-client.virtnet to discover 192.168.206.152@tcp (in background) [ 1433.918505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.152@tcp [ 1435.246438] Lustre: DEBUG MARKER: Force oleg652-server to discover 192.168.206.52@tcp [ 1436.960960] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1437.614599] Lustre: DEBUG MARKER: Wait for 71196 [ 1439.016789] Lustre: DEBUG MARKER: Finished wait on 71196 [ 1439.273748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1439.310791] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1439.318726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1439.618636] LNet: 71645:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.627239] LNetError: 71645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.646943] LNet: Removed LNI 192.168.206.52@tcp [ 1440.099689] Key type .llcrypt unregistered [ 1440.104915] Key type ._llcrypt unregistered [ 1455.867686] Key type ._llcrypt registered [ 1455.870425] Key type .llcrypt registered [ 1457.108620] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:31:00 (1788179460) [ 1457.611471] Key type .llcrypt unregistered [ 1457.614550] Key type ._llcrypt unregistered [ 1463.503261] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1463.749721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1463.758937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1463.766918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1466.321993] Key type ._llcrypt registered [ 1466.326048] Key type .llcrypt registered [ 1466.535458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.549339] alg: No test for adler32 (adler32-zlib) [ 1467.506654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.581463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1467.620921] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1468.478333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1468.506728] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1476.950502] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:31:20 (1788179480) [ 1477.050598] LNet: 73721:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.060470] LNetError: 73721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1477.066724] LNet: Removed LNI 10.1.2.3@tcp [ 1477.077523] LNet: Removed LNI 192.168.206.52@tcp [ 1477.818382] Key type .llcrypt unregistered [ 1477.820327] Key type ._llcrypt unregistered [ 1484.391785] Key type ._llcrypt registered [ 1484.404090] Key type .llcrypt registered [ 1484.723625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1484.748275] alg: No test for adler32 (adler32-zlib) [ 1485.672630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.740969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.784101] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1486.542892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1486.558567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1487.787171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.52@tcp [ 1488.864141] LNetError: Unexpected error -22 connecting to 192.168.206.52@tcp at host 192.168.206.52:988 [ 1495.698949] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:31:39 (1788179499) [ 1495.799091] LNet: 74846:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.811470] LNetError: 74846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1495.828086] LNet: Removed LNI 10.1.2.3@tcp [ 1495.849660] LNet: Removed LNI 192.168.206.52@tcp [ 1496.484599] Key type .llcrypt unregistered [ 1496.487161] Key type ._llcrypt unregistered [ 1503.356307] Key type ._llcrypt registered [ 1503.360872] Key type .llcrypt registered [ 1503.610716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1503.626674] alg: No test for adler32 (adler32-zlib) [ 1504.629491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.702390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1504.750515] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1505.888765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1505.915893] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 1506.681259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.52@tcp1 --nid 192.168.206.52@tcp2 [ 1510.879882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp1 192.168.206.52@tcp2 [ 1510.931741] LNetError: 75691:0:(lib-move.c:2066:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.52@tcp1 to 192.168.206.52@tcp2 - Route table may be misconfigured [ 1510.964645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp2 192.168.206.52@tcp1 [ 1517.219401] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:32:01 (1788179521) [ 1517.384299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.401489] LNet: 76265:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.406197] LNetError: 76265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.414484] LNet: Removed LNI 192.168.206.52@tcp1 [ 1518.444463] LNet: Removed LNI 192.168.206.52@tcp2 [ 1518.469714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.529483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.547351] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1519.219931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1519.243719] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1519.900492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 1519.936630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.52@tcp [ 1519.966810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp1 192.168.206.52@tcp1 [ 1519.998301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.52@tcp -d 192.168.206.52@tcp -e network_timeout [ 1520.038140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -e network_timeout [ 1520.080303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -e network_timeout [ 1520.104535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -e network_timeout [ 1520.137134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1520.161367] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1520.185193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1520.283883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1520.355259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1520.544083] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1525.626182] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:32:09 (1788179529) [ 1525.734538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.765713] LNet: 77119:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.776454] LNetError: 77119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.786764] LNetError: 77119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1525.794662] LNet: Removed LNI 192.168.206.52@tcp [ 1526.837800] LNet: Removed LNI 192.168.206.52@tcp1 [ 1526.884175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.995545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1527.230359] LNet: 77163:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.537901] Key type .llcrypt unregistered [ 1527.542558] Key type ._llcrypt unregistered [ 1537.405133] Key type ._llcrypt registered [ 1537.407613] Key type .llcrypt registered [ 1538.598826] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:32:22 (1788179542) [ 1539.108776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1539.128608] alg: No test for adler32 (adler32-zlib) [ 1540.037697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.506932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1542.847603] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:32:26 (1788179546) [ 1543.272899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.294863] LNet: 78277:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.320570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.411954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.434840] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1544.394745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1544.424007] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1545.624466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1545.669510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp1 [ 1545.717442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp1 [ 1551.876390] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:32:35 (1788179555) [ 1554.175410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg652-server [ 1555.765490] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:32:39 (1788179559) [ 1557.633424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg652-server [ 1559.020619] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:32:43 (1788179563) [ 1560.623855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg652-server [ 1562.232198] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:32:46 (1788179566) [ 1563.799806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg652-server [ 1565.492964] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:32:49 (1788179569) [ 1567.159351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg652-server [ 1568.608691] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:32:52 (1788179572) [ 1570.013265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg652-server [ 1571.857148] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:32:55 (1788179575) [ 1573.629390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg652-server [ 1575.184445] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:32:59 (1788179579) [ 1576.854185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg652-server [ 1578.217421] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:33:02 (1788179582) [ 1579.837498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg652-server [ 1581.316489] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:33:05 (1788179585) [ 1581.483437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.497984] LNet: 82154:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.506523] LNetError: 82154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.559286] LNet: Removed LNI 192.168.206.52@tcp [ 1582.573645] LNetError: 82154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.587878] LNet: Removed LNI 192.168.206.52@tcp1 [ 1582.607586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.641898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.667953] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1583.220419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1583.311580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1583.464147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.487193] LNet: 82301:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.555420] LNet: Removed LNI 192.168.206.52@tcp [ 1584.582125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.641930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1584.682745] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1585.347428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1585.483297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.499956] LNet: 82446:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.504523] LNetError: 82446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.509824] LNetError: 82446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1585.520681] LNet: Removed LNI 192.168.206.52@tcp [ 1585.534535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.581334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1585.607423] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1586.315314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1586.417451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1586.619553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.685631] LNet: Removed LNI 192.168.206.52@tcp [ 1587.722801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.775342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1587.800079] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1588.418808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1588.615884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.637589] LNetError: 82739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.644045] LNetError: 82739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1588.659914] LNet: Removed LNI 192.168.206.52@tcp [ 1588.690471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.741103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.774706] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1589.526537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1589.600685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1589.853541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.877501] LNet: 82886:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.887801] LNet: 82886:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1590.950096] LNet: Removed LNI 192.168.206.52@tcp [ 1590.979674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.030417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1591.060804] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1591.776787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1592.034177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.125744] LNet: Removed LNI 192.168.206.52@tcp [ 1593.156644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.210699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.232917] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1593.731327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1593.801055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1594.059608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.094278] LNetError: 83179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.100848] LNetError: 83179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1594.119917] LNet: Removed LNI 192.168.206.52@tcp [ 1594.134708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.167861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1594.191636] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1594.849274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1596.187537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.226242] LNet: Removed LNI 192.168.206.52@tcp [ 1597.268529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.353679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1597.382661] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1598.181343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1598.261457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1598.576348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.625444] LNet: 83480:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.632856] LNet: 83480:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1599.656716] LNet: Removed LNI 192.168.206.52@tcp [ 1599.701780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.775748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1599.795793] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1600.519850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1601.941457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.987232] LNet: Removed LNI 192.168.206.52@tcp [ 1603.027612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.091238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.129636] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1603.756353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1603.825747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1604.100411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.119389] LNetError: 83782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.123641] LNetError: 83782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1604.135276] LNet: Removed LNI 192.168.206.52@tcp [ 1604.175353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.254080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1604.278678] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1605.155773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1605.474640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.566141] LNet: Removed LNI 192.168.206.52@tcp [ 1606.625996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.693218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.718799] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1607.476652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1607.561447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.609712] LNet: Removed LNI 192.168.206.52@tcp [ 1607.628572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.680737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1607.708386] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1608.543237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1608.616353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.642961] LNet: Removed LNI 192.168.206.52@tcp [ 1608.672748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.739350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1608.818975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.850837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.899450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1609.012128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.072645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.150680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1609.232790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.272815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.325550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1609.351739] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1609.966996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.994096] LNet: Removed LNI 192.168.206.52@tcp [ 1610.018706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.087848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1617.026725] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:33:40 (1788179620) [ 1617.214722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.230946] LNet: 84948:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.235206] LNet: 84948:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1617.255943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.295339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1617.318498] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/180] [ 1618.439853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.463928] LNet: Removed LNI 192.168.206.52@tcp231 [ 1618.487630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.590026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1618.616274] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/180] [ 1618.627502] LNet: 85094:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.044643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1619.085962] LNet: Removed LNI 192.168.206.52@tcp231 [ 1619.106816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1619.143869] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/180] [ 1620.358612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.414578] LNetError: 85238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.428644] LNetError: 85238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1620.451831] LNet: Removed LNI 192.168.206.52@tcp231 [ 1620.472944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.589390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1620.612495] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/0] [ 1620.620874] LNet: 85246:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1621.266696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1621.314871] LNet: Removed LNI 192.168.206.52@tcp231 [ 1621.336964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1621.374663] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/0] [ 1622.410645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.456692] LNet: Removed LNI 192.168.206.52@tcp231 [ 1622.477608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.571351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1622.606187] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/60] [ 1622.613345] LNet: 85398:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1622.911459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1622.951178] LNet: Removed LNI 192.168.206.52@tcp231 [ 1622.972386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1623.004296] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/60] [ 1624.165032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.192563] LNet: Removed LNI 192.168.206.52@tcp231 [ 1624.215549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.290232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1624.324952] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/180] [ 1624.727721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1624.768375] LNet: Removed LNI 192.168.206.52@tcp231 [ 1624.788827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1624.836190] LNet: Added LNI 192.168.206.52@tcp231 [8/256/0/180] [ 1632.150780] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:33:55 (1788179635) [ 1632.458672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.507080] LNet: Removed LNI 192.168.206.52@tcp231 [ 1632.552129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.677428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1632.720657] LNet: Added LNI 192.168.206.52@tcp232 [8/256/0/180] [ 1633.667732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1633.719673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1633.798872] LNet: Removed LNI 192.168.206.52@tcp232 [ 1633.842818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1633.877925] LNet: Added LNI 192.168.206.52@tcp232 [8/256/0/180] [ 1634.868595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1634.936450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1634.981716] LNet: Removed LNI 192.168.206.52@tcp232 [ 1635.016234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1635.048820] LNet: Added LNI 192.168.206.52@tcp232 [8/256/0/180] [ 1635.724884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1635.781280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1635.818327] LNet: Removed LNI 192.168.206.52@tcp232 [ 1635.831510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1635.851669] LNet: Added LNI 192.168.206.52@tcp232 [8/256/0/180] [ 1636.644971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1636.723863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1636.768626] LNet: Removed LNI 192.168.206.52@tcp232 [ 1636.785198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1643.325599] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:34:06 (1788179646) [ 1644.632144] Key type .llcrypt unregistered [ 1644.634350] Key type ._llcrypt unregistered [ 1650.570581] Key type ._llcrypt registered [ 1650.577510] Key type .llcrypt registered [ 1650.778356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1650.790187] alg: No test for adler32 (adler32-zlib) [ 1651.708396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1651.732173] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1657.600484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1664.850527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.883554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1664.903429] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1665.923491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_interrupt -r 2 [ 1665.951671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_interrupt -r 2 [ 1665.984389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_interrupt -r 2 [ 1666.040158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_interrupt -r 2 [ 1666.078790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_interrupt -r 2 [ 1666.113655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_interrupt -r 2 [ 1666.151190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_interrupt -r 2 [ 1666.192634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_interrupt -r 2 [ 1666.233434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.152@tcp [ 1666.259122] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1666.271047] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1666.448765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1666.472854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1666.496600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1666.527408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1666.565604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1666.593606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1666.635805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1666.678255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1666.700220] LNet: 88048:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1666.724475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1666.766538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1666.812307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1666.858191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1666.906765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1666.943362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1667.382450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_dropped -r 2 [ 1667.432906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_dropped -r 2 [ 1667.475613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_dropped -r 2 [ 1667.509652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_dropped -r 2 [ 1667.539055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_dropped -r 2 [ 1667.566060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_dropped -r 2 [ 1667.591404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_dropped -r 2 [ 1667.623506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_dropped -r 2 [ 1667.654711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.152@tcp [ 1667.678635] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1667.877338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1667.913533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1667.938217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1667.981700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1668.017454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1668.043250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1668.071174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1668.092653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1668.106407] LNet: 88168:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1668.121822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1668.149960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1668.170748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1668.199612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1668.233288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1668.268332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1668.660616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_aborted -r 2 [ 1668.694967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_aborted -r 2 [ 1668.736984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_aborted -r 2 [ 1668.783514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_aborted -r 2 [ 1668.833601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_aborted -r 2 [ 1668.880447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_aborted -r 2 [ 1668.930764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_aborted -r 2 [ 1668.984510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_aborted -r 2 [ 1669.026345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.152@tcp [ 1669.047203] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1669.249303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1669.283641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1669.312659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1669.336770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1669.355608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1669.381847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1669.413604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1669.450690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1669.466853] LNet: 88288:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1669.488471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1669.517959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1669.543690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1669.577436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1669.611994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1669.640977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1670.069452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_no_route -r 2 [ 1670.099781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_no_route -r 2 [ 1670.136613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_no_route -r 2 [ 1670.176985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_no_route -r 2 [ 1670.220129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_no_route -r 2 [ 1670.259123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_no_route -r 2 [ 1670.297357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_no_route -r 2 [ 1670.329273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_no_route -r 2 [ 1670.373229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.152@tcp [ 1670.595579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.615972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1670.641587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.672427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1670.704264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1670.750272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1670.778667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1670.811638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1670.849534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1670.875541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1670.904635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1670.929894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1670.959948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1670.997758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1671.512471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e local_timeout -r 2 [ 1671.542607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e local_timeout -r 2 [ 1671.574333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e local_timeout -r 2 [ 1671.606287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e local_timeout -r 2 [ 1671.640449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e local_timeout -r 2 [ 1671.694332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e local_timeout -r 2 [ 1671.742739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e local_timeout -r 2 [ 1671.790613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e local_timeout -r 2 [ 1671.845837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.152@tcp [ 1671.871769] LNet: There was an unexpected network error while writing to 192.168.206.152: rc = -22 [ 1671.879853] LNet: Skipped 1 previous similar message [ 1672.128849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1672.164724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1672.209686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1672.243023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1672.293382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1672.339327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1672.389326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1672.441746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1672.483870] LNet: 88529:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1672.490341] LNet: 88529:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1672.511589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1672.547584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1672.589962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1672.620284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1672.646449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1672.696664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1673.161303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp -m GET -e remote_dropped -r 2 [ 1673.190900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp -d 192.168.206.152@tcp1 -m GET -e remote_dropped -r 2 [ 1673.239565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp -m GET -e remote_dropped -r 2 [ 1673.282463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp -d 192.168.206.52@tcp1 -m GET -e remote_dropped -r 2 [ 1673.317516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp -m GET -e remote_dropped -r 2 [ 1673.350723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.52@tcp1 -d 192.168.206.152@tcp1 -m GET -e remote_dropped -r 2 [ 1673.395466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp -m GET -e remote_dropped -r 2 [ 1673.431970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.52@tcp1 -d 192.168.206.52@tcp1 -m GET -e remote_dropped -r 2 [ 1673.471554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.52@tcp 192.168.206.152@tcp [ 1673.495954] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.152@tcp [ 1673.719790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1673.741784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1673.764552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1673.786591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1673.816360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1673.840298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1673.871817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1673.897304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1673.932268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1673.957885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1673.983786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1674.012110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1674.059663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1674.099767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1678.497426] LNet: 88723:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.507768] LNetError: 88723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.519282] LNet: Removed LNI 192.168.206.52@tcp [ 1679.520993] LNetError: 88723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.544489] LNet: Removed LNI 192.168.206.52@tcp1 [ 1679.898750] Key type .llcrypt unregistered [ 1679.904688] Key type ._llcrypt unregistered [ 1694.516847] Key type ._llcrypt registered [ 1694.522093] Key type .llcrypt registered [ 1695.694211] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:34:59 (1788179699) [ 1696.113609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.135601] alg: No test for adler32 (adler32-zlib) [ 1697.089634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.157791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.184889] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1697.942410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1697.968607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.52@tcp [ 1697.985336] LNetError: 89701:0:(api-ni.c:4475:lnet_ni_set_healthv()) manually adding local NI 192.168.206.52@tcp to recovery [ 1698.083718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.52@tcp [ 1698.180404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1703.945765] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:35:07 (1788179707) [ 1704.065393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.083899] LNet: 90285:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.088870] LNetError: 90285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.100893] LNet: Removed LNI 192.168.206.52@tcp [ 1704.125646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.208736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.230562] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1704.893447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1704.919559] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1705.730700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.52@tcp [ 1705.751235] LNetError: 90557:0:(api-ni.c:4475:lnet_ni_set_healthv()) manually adding local NI 192.168.206.52@tcp to recovery [ 1706.207495] LNet: 1 local NIs in recovery (showing 1): 192.168.206.52@tcp [ 1706.217939] LNet: There was an unexpected network error while writing to 192.168.206.52: rc = -22 [ 1706.990721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.52@tcp [ 1715.479645] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:35:19 (1788179719) [ 1717.472832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg652-server [ 1719.103181] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:35:22 (1788179722) [ 1721.020646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg652-server [ 1723.225365] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:35:26 (1788179726) [ 1725.135906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg652-server [ 1726.866344] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:35:30 (1788179730) [ 1727.037994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.073591] LNet: 92189:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.084462] LNetError: 92189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.088638] LNet: Removed LNI 192.168.206.52@tcp [ 1727.123261] LNet: Removed LNI 192.168.206.52@tcp1 [ 1727.138926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.181619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.224471] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1727.941656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1728.025716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1728.066970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1728.145163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1728.194078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1728.225057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1733.837370] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:35:37 (1788179737) [ 1734.016649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.054477] LNet: 92914:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.062550] LNetError: 92914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.069898] LNetError: 92914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1734.083938] LNet: Removed LNI 192.168.206.52@tcp [ 1734.112809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.188349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.218425] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1740.758946] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:35:44 (1788179744) [ 1740.869085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.913374] LNet: 93611:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.927233] LNetError: 93611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.955452] LNet: Removed LNI 192.168.206.52@tcp [ 1741.005645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.095344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.136595] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1749.143786] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:35:51 (1788179751) [ 1751.241147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1752.853678] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:35:56 (1788179756) [ 1753.364133] LNet: 94653:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.369506] LNetError: 94653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.393712] LNet: Removed LNI 192.168.206.52@tcp [ 1754.181900] Key type .llcrypt unregistered [ 1754.184394] Key type ._llcrypt unregistered [ 1760.766498] Key type ._llcrypt registered [ 1760.769964] Key type .llcrypt registered [ 1761.077929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1761.094799] alg: No test for adler32 (adler32-zlib) [ 1762.053530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1762.088923] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1769.117850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1772.439565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.152@tcp [ 1772.733151] LNet: 95084:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.737239] LNetError: 95084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.750786] LNet: Removed LNI 192.168.206.52@tcp [ 1773.137460] Key type .llcrypt unregistered [ 1773.142435] Key type ._llcrypt unregistered [ 1782.422754] Key type ._llcrypt registered [ 1782.429335] Key type .llcrypt registered [ 1783.786936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1785.184164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1786.854514] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:36:30 (1788179790) [ 1788.517730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1790.182188] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:36:34 (1788179794) [ 1791.775900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg652-server [ 1793.182199] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:36:37 (1788179797) [ 1794.984599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg652-server [ 1796.583346] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:36:40 (1788179800) [ 1797.937772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.958155] alg: No test for adler32 (adler32-zlib) [ 1799.033128] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1799.218255] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1800.855475] Key type lgssc registered [ 1801.938985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.140304] Key type lgssc unregistered [ 1819.436503] LNet: 98259:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.443241] LNetError: 98259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.463305] LNet: Removed LNI 192.168.206.52@tcp [ 1820.083585] Key type .llcrypt unregistered [ 1820.090562] Key type ._llcrypt unregistered [ 1833.354682] Key type ._llcrypt registered [ 1833.356478] Key type .llcrypt registered [ 1834.566283] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:37:18 (1788179838) [ 1835.040695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.051164] alg: No test for adler32 (adler32-zlib) [ 1835.967302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.018707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1836.051496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1836.082073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1836.110960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1841.005910] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:37:24 (1788179844) [ 1841.100663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.128482] LNet: 99668:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.151653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.208740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1841.260462] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1842.212973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.238394] LNet: 99819:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.246714] LNetError: 99819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.268587] LNet: Removed LNI 192.168.206.52@tcp [ 1842.292633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.361497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1842.396526] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1842.879519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.893392] LNetError: 99847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.909613] LNet: Removed LNI 192.168.206.52@tcp [ 1842.967221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.027358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1843.051479] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1849.043153] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:37:32 (1788179852) [ 1849.321451] LNet: 100436:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.339181] LNet: 100436:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1849.354488] LNetError: 100436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.367667] LNet: Removed LNI 192.168.206.52@tcp [ 1849.909513] Key type .llcrypt unregistered [ 1849.911681] Key type ._llcrypt unregistered [ 1856.293676] Key type ._llcrypt registered [ 1856.296205] Key type .llcrypt registered [ 1856.496196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1856.522305] alg: No test for adler32 (adler32-zlib) [ 1857.460079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.555916] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1857.558312] LNet: *** cfs_fail_loc=e003, val=100*** [ 1858.324458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1858.355290] LNet: *** cfs_fail_loc=e003, val=100*** [ 1858.358851] LNet: Skipped 2 previous similar messages [ 1858.365691] Ping reply received with mapping: [ 1858.373852] found metadata reply NI[192.168.206.52@tcp] [ 1858.378094] 34 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1858.380871] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1858.385730] 5a 5a 5a 5a 5a 5a 5a 5a [ 1858.573399] LNet: 100834:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.584303] LNetError: 100834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.620952] LNet: Removed LNI 192.168.206.52@tcp [ 1860.409378] Key type .llcrypt unregistered [ 1860.410851] Key type ._llcrypt unregistered [ 1871.193084] Key type ._llcrypt registered [ 1871.194629] Key type .llcrypt registered [ 1872.908938] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:37:56 (1788179876) [ 1873.575572] Key type .llcrypt unregistered [ 1873.578549] Key type ._llcrypt unregistered [ 1879.972675] Key type ._llcrypt registered [ 1879.976915] Key type .llcrypt registered [ 1880.387775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.426341] alg: No test for adler32 (adler32-zlib) [ 1881.832889] LNetError: 101881:0:(api-ni.c:2842:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1882.418452] Key type .llcrypt unregistered [ 1882.424069] Key type ._llcrypt unregistered [ 1888.456883] Key type ._llcrypt registered [ 1888.459771] Key type .llcrypt registered [ 1889.924315] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:38:13 (1788179893) [ 1890.731242] Key type .llcrypt unregistered [ 1890.734328] Key type ._llcrypt unregistered [ 1896.622032] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1896.643810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1899.262267] Key type ._llcrypt registered [ 1899.264901] Key type .llcrypt registered [ 1899.484461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1899.510310] alg: No test for adler32 (adler32-zlib) [ 1900.489745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.586120] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1910.419852] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:38:34 (1788179914) [ 1910.555782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.592317] LNet: 103657:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.608488] LNetError: 103657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1910.644985] LNet: Removed LNI 10.1.2.3@tcp [ 1910.688753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.680768] Key type .llcrypt unregistered [ 1911.683587] Key type ._llcrypt unregistered [ 1918.711626] Key type ._llcrypt registered [ 1918.715028] Key type .llcrypt registered [ 1918.924484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1918.942330] alg: No test for adler32 (adler32-zlib) [ 1919.973179] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1920.094814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1920.338817] LNet: 103951:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.344806] LNetError: 103951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.367538] LNet: Removed LNI 192.168.206.52@tcp [ 1921.105404] Key type .llcrypt unregistered [ 1921.107547] Key type ._llcrypt unregistered [ 1931.998611] Key type ._llcrypt registered [ 1932.001444] Key type .llcrypt registered [ 1933.164100] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:38:57 (1788179937) [ 1933.702354] Key type .llcrypt unregistered [ 1933.705105] Key type ._llcrypt unregistered [ 1939.973807] Key type ._llcrypt registered [ 1939.978731] Key type .llcrypt registered [ 1940.289665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1940.312072] alg: No test for adler32 (adler32-zlib) [ 1947.255343] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:39:11 (1788179951) [ 1947.509959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.542838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.616391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.658284] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1948.590418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.648516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1948.673744] LNet: Removed LNI 192.168.206.52@tcp [ 1948.961585] LNet: 105696:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.309405] Key type .llcrypt unregistered [ 1949.311424] Key type ._llcrypt unregistered [ 1959.712395] Key type ._llcrypt registered [ 1959.716451] Key type .llcrypt registered [ 1960.990634] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:39:24 (1788179964) [ 1961.587716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.601970] alg: No test for adler32 (adler32-zlib) [ 1962.596078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.659058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.687663] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1963.457741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 1963.494736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.52@tcp [ 1963.511538] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1963.516665] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1963.519397] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1963, 0, 0 [ 1963.524634] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1963.527663] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1963.537238] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1963.541343] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1963, 0, 0 [ 1963.544209] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1963.550740] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1963.554307] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1963.557482] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1963, 0, 2013 [ 1963.561409] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1963.563710] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1963.566342] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1963.568802] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1963, 0, 2013 [ 1963.573035] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1970.955438] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:39:34 (1788179974) [ 1971.306993] LNet: 107236:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.313539] LNetError: 107236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.335215] LNet: Removed LNI 192.168.206.52@tcp [ 1971.931735] Key type .llcrypt unregistered [ 1971.942743] Key type ._llcrypt unregistered [ 1978.009567] Key type ._llcrypt registered [ 1978.014762] Key type .llcrypt registered [ 1978.229520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1978.249763] alg: No test for adler32 (adler32-zlib) [ 1979.229915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1979.271762] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 1986.844734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 1996.842162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.910786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1996.938332] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 1998.194123] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1998.342201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1998.357812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1998.370604] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2001.099987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2001.135162] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2010.868675] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:40:14 (1788180014) [ 2011.016163] LNet: 108741:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.025459] LNetError: 108741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.041730] LNet: Removed LNI 192.168.206.52@tcp [ 2012.064750] LNet: Removed LNI 192.168.206.52@tcp1 [ 2012.067255] LNetError: 108741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2012.072621] LNetError: 108741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2012.090358] LNet: Removed LNI 10.1.2.3@tcp99 [ 2012.736895] Key type .llcrypt unregistered [ 2012.738605] Key type ._llcrypt unregistered [ 2017.789071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2017.804377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2020.347561] Key type ._llcrypt registered [ 2020.355228] Key type .llcrypt registered [ 2020.631894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.650386] alg: No test for adler32 (adler32-zlib) [ 2021.632425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.775491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.802315] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2022.822953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2022.870485] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2024.950625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2025.033350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2025.081981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2025.122572] LNetError: 109490:0:(peer.c:2516:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2025.164453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2025.227181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2025.714426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2025.770279] LNet: 109510:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.779122] LNetError: 109510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2025.784860] LNet: Removed LNI 10.1.2.3@tcp [ 2026.854970] LNet: Removed LNI 192.168.206.52@tcp [ 2027.647675] Key type .llcrypt unregistered [ 2027.655082] Key type ._llcrypt unregistered [ 2038.652996] Key type ._llcrypt registered [ 2038.655826] Key type .llcrypt registered [ 2040.251587] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:40:43 (1788180043) [ 2040.720308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2040.731527] alg: No test for adler32 (adler32-zlib) [ 2041.624822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.703233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2041.727870] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2042.490277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 2042.572079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg652-client@tcp [ 2048.291841] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:40:52 (1788180052) [ 2048.523228] LNet: 111043:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.531885] LNetError: 111043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.546272] LNet: Removed LNI 192.168.206.52@tcp [ 2049.039577] Key type .llcrypt unregistered [ 2049.041557] Key type ._llcrypt unregistered [ 2055.441951] Key type ._llcrypt registered [ 2055.446356] Key type .llcrypt registered [ 2055.665324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.676219] alg: No test for adler32 (adler32-zlib) [ 2056.579604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2056.614798] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2056.651922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2062.460518] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:41:06 (1788180066) [ 2064.088573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2065.841054] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:41:09 (1788180069) [ 2067.126817] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2068.588940] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:41:12 (1788180072) [ 2068.697231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2068.731371] LNet: 112461:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.739297] LNetError: 112461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.770432] LNet: Removed LNI 192.168.206.52@tcp [ 2068.784746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.829514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2068.862741] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2069.451242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2069.508977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2075.995646] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:41:19 (1788180079) [ 2076.170536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.194477] LNet: 113175:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.200359] LNetError: 113175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.215103] LNet: Removed LNI 192.168.206.52@tcp [ 2076.235297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.277424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2076.312186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2076.928383] Key type .llcrypt unregistered [ 2076.931200] Key type ._llcrypt unregistered [ 2087.782246] Key type ._llcrypt registered [ 2087.784483] Key type .llcrypt registered [ 2088.928584] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:41:32 (1788180092) [ 2089.327265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.340574] alg: No test for adler32 (adler32-zlib) [ 2090.318467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.385252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2090.420885] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2091.198844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2091.251245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2098.056544] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:41:41 (1788180101) [ 2098.172487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.199034] LNet: 114726:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.203760] LNetError: 114726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.219938] LNet: Removed LNI 192.168.206.52@tcp [ 2098.287223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.360974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2098.413823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2104.587891] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:41:48 (1788180108) [ 2104.776439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.792670] LNet: 115311:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.807265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.858321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2105.235871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2105.290465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.377397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2105.758869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2105.801464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.905981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.964986] LNet: 115345:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.997927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.077359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2106.613570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2106.652788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2106.718433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2106.983235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2107.012042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2113.283667] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:41:57 (1788180117) [ 2113.445607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.479851] LNet: 115945:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.498844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.557562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.598129] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2114.174379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2114.193661] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2114.885607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2114.938169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2115.118609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.136796] LNetError: 116224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.191167] LNet: Removed LNI 192.168.206.52@tcp [ 2116.195606] LNetError: 116224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.215896] LNet: Removed LNI 192.168.206.52@tcp2 [ 2116.233881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.306774] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2116.312141] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2116.317575] LNet: 116231:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2116.810974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2117.003830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.048930] LNet: Removed LNI 192.168.206.52@tcp [ 2118.070443] LNet: Removed LNI 192.168.206.52@tcp2 [ 2118.131621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.202859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.223288] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2118.860486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2118.881498] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2119.831218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2119.878886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2119.943593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2120.201854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.219368] LNet: 116552:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.223823] LNet: 116552:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2120.230713] LNetError: 116552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.236608] LNetError: 116552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2120.241902] LNet: Removed LNI 192.168.206.52@tcp [ 2121.261394] LNet: Removed LNI 192.168.206.52@tcp2 [ 2121.291199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.383649] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2121.391001] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2121.396708] LNet: 116559:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2121.851929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2122.075791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.095827] LNet: Removed LNI 192.168.206.52@tcp [ 2123.109866] LNet: Removed LNI 192.168.206.52@tcp2 [ 2123.132381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.203960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.257569] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2124.007416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2124.039522] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2125.032600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2125.100805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2125.295298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.317328] LNetError: 116875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.322497] LNetError: 116875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2125.326596] LNet: Removed LNI 192.168.206.52@tcp [ 2125.337948] LNet: Removed LNI 192.168.206.52@tcp2 [ 2125.361523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.416500] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2125.421332] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2125.428872] LNet: 116881:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2126.149720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2126.349338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.392375] LNet: Removed LNI 192.168.206.52@tcp [ 2127.401830] LNet: Removed LNI 192.168.206.52@tcp2 [ 2127.426388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.500553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.539864] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2128.265652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2128.288138] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2129.151321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2129.190957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2129.281224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2129.532536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.553705] LNet: 117203:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.560967] LNet: 117203:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2130.591238] LNet: Removed LNI 192.168.206.52@tcp [ 2131.624290] LNet: Removed LNI 192.168.206.52@tcp2 [ 2131.665133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.736718] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2131.743063] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2131.753098] LNet: 117210:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.276760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2132.486404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.508749] LNet: Removed LNI 192.168.206.52@tcp [ 2133.545412] LNet: Removed LNI 192.168.206.52@tcp2 [ 2133.583325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.642770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.664602] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2134.375551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2134.406526] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2135.127385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2135.160903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2135.217152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2135.544054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.566818] LNetError: 117543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.579130] LNetError: 117543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2135.587522] LNet: Removed LNI 192.168.206.52@tcp [ 2135.602143] LNet: Removed LNI 192.168.206.52@tcp2 [ 2135.631316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.716055] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2135.726359] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2136.238946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2136.615747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.695307] LNet: Removed LNI 192.168.206.52@tcp [ 2137.707853] LNet: Removed LNI 192.168.206.52@tcp2 [ 2137.742542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.800676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.821902] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2138.552656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2138.579692] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2139.308884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2139.346925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2139.374673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2139.427636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2139.718271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.767196] LNet: Removed LNI 192.168.206.52@tcp [ 2141.798739] LNet: Removed LNI 192.168.206.52@tcp2 [ 2141.850946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.931881] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2141.956725] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2141.976775] LNet: 117910:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2141.989521] LNet: 117910:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2142.691991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2148.867819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2150.369351] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:42:34 (1788180154) [ 2150.455931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.486321] LNet: 118661:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.490396] LNet: 118661:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2150.498827] LNet: Removed LNI 192.168.206.52@tcp [ 2151.525867] LNet: Removed LNI 192.168.206.52@tcp2 [ 2151.566465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.979940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.016460] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2152.721564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2152.739686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2153.430176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.474320] LNetError: 118944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2153.480937] LNetError: 118944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2153.490460] LNet: Removed LNI 10.1.2.3@tcp [ 2153.505423] LNet: Removed LNI 192.168.206.52@tcp [ 2153.525801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.581908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2153.628540] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2153.635374] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2154.049513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2154.066102] LNet: Removed LNI 192.168.206.52@tcp [ 2154.079334] LNet: Removed LNI 10.1.2.3@tcp [ 2159.567341] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:42:43 (1788180163) [ 2159.725385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.811212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.928217] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2160.070021] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2160.078168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2160.086453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2162.557174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2162.588399] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2163.406871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2163.425196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2165.427596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp [ 2165.488662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2165.539550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.52@tcp -m PUT -e local_timeout -r 1 [ 2165.602434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2165.683257] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2165.686466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2165.980384] LNetError: 119609:0:(lib-move.c:4773:lnet_parse()) 192.168.206.52@tcp, src 192.168.206.52@tcp: Dropping PUT (error -108 looking up sender) [ 2167.009463] LNet: Removed LNI 10.1.2.3@tcp [ 2168.045284] LNet: Removed LNI 192.168.206.52@tcp [ 2168.614853] Key type .llcrypt unregistered [ 2168.621581] Key type ._llcrypt unregistered [ 2182.941303] Key type ._llcrypt registered [ 2182.944664] Key type .llcrypt registered [ 2183.919960] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:43:07 (1788180187) [ 2184.535398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.549572] alg: No test for adler32 (adler32-zlib) [ 2185.516592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.882414] LNet: 120921:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.512619] Key type .llcrypt unregistered [ 2186.517344] Key type ._llcrypt unregistered [ 2192.700890] Key type ._llcrypt registered [ 2192.703690] Key type .llcrypt registered [ 2192.916438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2192.941836] alg: No test for adler32 (adler32-zlib) [ 2193.892780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2193.915215] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2202.167717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 2202.447500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2203.293270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.152@tcp [ 2203.512916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2203.538567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2203.564472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2203.618554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2203.641574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2203.672158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2203.728747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2203.787674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2203.821786] LNet: 121372:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2203.852094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2203.941438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2203.998183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2204.035360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2204.074951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2204.097603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.634473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2204.868736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2204.895970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2204.934199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2204.983199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2205.020711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2205.069983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2205.099958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2205.125520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2205.144437] LNet: 121489:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2205.160781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2205.204306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2205.215331] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.152@tcp [ 2205.236692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2205.265445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2205.301341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2205.331480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2205.985971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.152@tcp [ 2206.280867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2206.342348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2206.405570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2206.447165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2206.474296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2206.504617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2206.540447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2206.605405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2206.636285] LNet: 121606:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.659422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2206.692844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2206.731640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2206.781855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2206.827983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2206.860929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2207.584733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2207.910921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2207.961680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2208.031025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2208.096859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2208.162335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2208.243786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2208.296589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2208.334448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2208.371519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2208.401756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2208.430131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2208.466443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2208.516208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2208.668365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2209.285640] LNet: 121770:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.301190] LNetError: 121770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.343914] LNet: Removed LNI 192.168.206.52@tcp [ 2209.901922] Key type .llcrypt unregistered [ 2209.913844] Key type ._llcrypt unregistered [ 2226.832746] Key type ._llcrypt registered [ 2226.838141] Key type .llcrypt registered [ 2228.255683] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:43:51 (1788180231) [ 2228.700984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2228.719637] alg: No test for adler32 (adler32-zlib) [ 2229.682682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.082409] LNet: 122659:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.909429] Key type .llcrypt unregistered [ 2230.917072] Key type ._llcrypt unregistered [ 2237.724244] Key type ._llcrypt registered [ 2237.726554] Key type .llcrypt registered [ 2237.914205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2237.931101] alg: No test for adler32 (adler32-zlib) [ 2238.821464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2238.850391] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2246.704787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.152@tcp [ 2256.296659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.346707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2256.370908] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2257.599775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2258.406856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.152@tcp [ 2258.671737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2258.706858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2258.742168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2258.772888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2258.801596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2258.841805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2258.875747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2258.904758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2258.920667] LNet: 123312:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2258.938483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2258.964624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2258.996568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2259.031730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2259.069313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2259.122304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2259.789833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2260.061451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2260.118278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2260.161439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2260.200141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2260.239584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2260.285763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2260.338828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2260.385897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2260.387612] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.152@tcp, 192.168.206.152@tcp1 [ 2260.426849] LNet: 123430:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2260.452066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2260.499137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2260.570866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2260.609125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2260.645714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2260.697032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2261.421931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.152@tcp [ 2261.700051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2261.753389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2261.816436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2261.862158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2261.891790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2261.921867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2261.960381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2261.995964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2262.010856] LNet: 123547:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2262.026667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2262.067424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2262.096257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2262.152087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2262.189742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2262.230827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2262.982976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2263.306165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2263.340558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2263.376909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2263.420860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2263.448626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2263.509896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2263.560235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2263.601141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2263.659254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2263.691347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2263.738686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2263.805901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2263.851845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2263.915648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2269.246491] LNet: 123745:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.262418] LNetError: 123745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.273584] LNet: Removed LNI 192.168.206.52@tcp [ 2269.292329] LNet: Removed LNI 192.168.206.52@tcp1 [ 2269.779519] Key type .llcrypt unregistered [ 2269.781720] Key type ._llcrypt unregistered [ 2284.895934] Key type ._llcrypt registered [ 2284.903525] Key type .llcrypt registered [ 2286.294883] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:44:49 (1788180289) [ 2287.860256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2289.716957] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:44:53 (1788180293) [ 2297.299227] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:45:01 (1788180301) [ 2304.228380] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:45:08 (1788180308) [ 2311.472511] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:45:15 (1788180315) [ 2318.522470] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:45:22 (1788180322) [ 2326.545503] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:45:29 (1788180329) [ 2334.803649] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:45:38 (1788180338) [ 2343.390696] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:45:46 (1788180346) [ 2351.326938] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:45:55 (1788180355) [ 2357.709358] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:46:01 (1788180361) [ 2365.606219] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:46:09 (1788180369) [ 2373.349316] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:46:17 (1788180377) [ 2381.178556] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:46:24 (1788180384) [ 2389.199776] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:46:32 (1788180392) [ 2397.364500] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:46:40 (1788180400) [ 2405.022253] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:46:48 (1788180408) [ 2412.471860] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:46:56 (1788180416) [ 2419.014850] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:47:02 (1788180422) [ 2419.341114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.359162] alg: No test for adler32 (adler32-zlib) [ 2420.199360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.329233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.354247] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2421.337344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.350509] LNet: 134745:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.355220] LNetError: 134745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.364811] LNet: Removed LNI 192.168.206.52@tcp [ 2421.376763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.417719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2421.438161] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2428.740304] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:47:11 (1788180431) [ 2437.832026] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:47:21 (1788180441) [ 2447.080196] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:47:30 (1788180450) [ 2453.682984] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:47:37 (1788180457) [ 2459.427724] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:47:43 (1788180463) [ 2466.994870] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:47:50 (1788180470) [ 2474.513858] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:47:58 (1788180478) [ 2481.079462] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:48:04 (1788180484) [ 2488.423750] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:48:11 (1788180491) [ 2497.228808] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:48:20 (1788180500) [ 2506.087574] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:48:29 (1788180509) [ 2512.253964] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:48:36 (1788180516) [ 2518.482217] Lustre: DEBUG MARKER: == sanity-lnet test 630: self-only view advertises only the ingress NI's NID ========================================================== 08:48:42 (1788180522) [ 2518.561902] LNet: 142204:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.574151] LNetError: 142204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.593334] LNet: Removed LNI 192.168.206.52@tcp [ 2519.133534] Key type .llcrypt unregistered [ 2519.138418] Key type ._llcrypt unregistered [ 2525.444754] Key type ._llcrypt registered [ 2525.449752] Key type .llcrypt registered [ 2525.697427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2525.707203] alg: No test for adler32 (adler32-zlib) [ 2526.640799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2526.713426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2526.751282] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2527.336650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2527.356666] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2528.244649] LNet: 142738:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.250298] LNetError: 142738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.258488] LNet: Removed LNI 192.168.206.52@tcp [ 2529.322052] LNet: Removed LNI 192.168.206.52@tcp1 [ 2530.000702] Key type .llcrypt unregistered [ 2530.004736] Key type ._llcrypt unregistered [ 2543.229865] Key type ._llcrypt registered [ 2543.236293] Key type .llcrypt registered [ 2545.002643] Lustre: DEBUG MARKER: == sanity-lnet test 631: self-only view keeps a node's NIDs as separate peers ========================================================== 08:49:08 (1788180548) [ 2545.582542] Key type .llcrypt unregistered [ 2545.587041] Key type ._llcrypt unregistered [ 2550.740916] Key type ._llcrypt registered [ 2550.744888] Key type .llcrypt registered [ 2550.970285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2550.992182] alg: No test for adler32 (adler32-zlib) [ 2551.864961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2551.936449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2551.979920] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2552.602969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2552.618117] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2553.867942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 2553.951120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp1 [ 2553.990544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp1 [ 2554.145809] LNet: 144030:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.157621] LNetError: 144030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.162809] LNet: Removed LNI 192.168.206.52@tcp [ 2555.240133] LNet: Removed LNI 192.168.206.52@tcp1 [ 2556.205163] Key type .llcrypt unregistered [ 2556.214126] Key type ._llcrypt unregistered [ 2569.031958] Key type ._llcrypt registered [ 2569.035938] Key type .llcrypt registered [ 2570.456423] Lustre: DEBUG MARKER: == sanity-lnet test 632: numa group view coalesces a node's NUMA-local NIDs ========================================================== 08:49:33 (1788180573) [ 2570.917488] Key type .llcrypt unregistered [ 2570.919967] Key type ._llcrypt unregistered [ 2577.077298] Key type ._llcrypt registered [ 2577.079612] Key type .llcrypt registered [ 2577.195586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2577.216933] alg: No test for adler32 (adler32-zlib) [ 2578.125550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2578.185976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2578.215499] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2578.219722] LNet: *** cfs_fail_loc=e005, val=2*** [ 2579.013179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2579.027568] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2579.030541] LNet: *** cfs_fail_loc=e005, val=2*** [ 2580.131886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp [ 2580.183309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.52@tcp1 [ 2580.212958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp1 [ 2580.271500] LNet: 145325:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.276398] LNetError: 145325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.290547] LNet: Removed LNI 192.168.206.52@tcp [ 2581.349528] LNet: Removed LNI 192.168.206.52@tcp1 [ 2582.311153] Key type .llcrypt unregistered [ 2582.313633] Key type ._llcrypt unregistered [ 2588.813750] Key type ._llcrypt registered [ 2588.816083] Key type .llcrypt registered [ 2588.963595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2588.989695] alg: No test for adler32 (adler32-zlib) [ 2589.903648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2589.995302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2590.035629] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2590.040573] LNet: *** cfs_fail_loc=e005, val=0*** [ 2590.546651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2590.565590] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2590.570119] LNet: *** cfs_fail_loc=e005, val=0*** [ 2591.402326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp1 [ 2591.483421] LNet: 145851:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.487166] LNetError: 145851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.491086] LNet: Removed LNI 192.168.206.52@tcp [ 2592.550995] LNet: Removed LNI 192.168.206.52@tcp1 [ 2593.119938] Key type .llcrypt unregistered [ 2593.122738] Key type ._llcrypt unregistered [ 2605.504157] Key type ._llcrypt registered [ 2605.506715] Key type .llcrypt registered [ 2606.979924] Lustre: DEBUG MARKER: == sanity-lnet test 633: numa grouped NID never leaks the merged full view ========================================================== 08:50:10 (1788180610) [ 2607.762172] Key type .llcrypt unregistered [ 2607.763689] Key type ._llcrypt unregistered [ 2615.362186] Key type ._llcrypt registered [ 2615.364237] Key type .llcrypt registered [ 2615.525519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2615.533619] alg: No test for adler32 (adler32-zlib) [ 2616.486207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2616.558911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2616.584485] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2616.591719] LNet: *** cfs_fail_loc=e005, val=0*** [ 2617.463815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2617.479808] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2617.482709] LNet: *** cfs_fail_loc=e005, val=0*** [ 2618.373713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2618.407638] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2619.804667] LNet: 147283:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.809257] LNetError: 147283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.814824] LNet: Removed LNI 192.168.206.52@tcp [ 2619.819872] LNet: Removed LNI 192.168.206.52@tcp1 [ 2620.839595] LNet: Removed LNI 192.168.206.52@tcp2 [ 2621.694733] Key type .llcrypt unregistered [ 2621.708134] Key type ._llcrypt unregistered [ 2635.281248] Key type ._llcrypt registered [ 2635.283584] Key type .llcrypt registered [ 2636.842726] Lustre: DEBUG MARKER: == sanity-lnet test 634: numa Push falls back when one net carries two groups ========================================================== 08:50:40 (1788180640) [ 2639.051445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_634 Need two interfaces to put two numa groups on one net [ 2641.418436] Lustre: DEBUG MARKER: == sanity-lnet test 635: numa Push sources from the group member on the peer's net ========================================================== 08:50:44 (1788180644) [ 2641.885552] Key type .llcrypt unregistered [ 2641.887451] Key type ._llcrypt unregistered [ 2648.912732] Key type ._llcrypt registered [ 2648.919587] Key type .llcrypt registered [ 2649.241467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2649.273771] alg: No test for adler32 (adler32-zlib) [ 2650.267642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2650.365485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2650.407217] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2650.414437] LNet: *** cfs_fail_loc=e005, val=0*** [ 2651.143687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2651.165891] LNet: Added LNI 192.168.206.52@tcp1 [8/256/0/180] [ 2651.169294] LNet: *** cfs_fail_loc=e005, val=0*** [ 2652.008215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 192.168.206.52@tcp1 [ 2652.075542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.52@tcp1 --state 1 [ 2652.105594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2652.120928] LNet: Added LNI 192.168.206.52@tcp2 [8/256/0/180] [ 2658.007191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.52@tcp1 [ 2658.111726] LNet: 149004:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2658.123228] LNetError: 149004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2658.133230] LNet: Removed LNI 192.168.206.52@tcp [ 2659.167232] LNet: Removed LNI 192.168.206.52@tcp1 [ 2659.172110] LNetError: 149004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.182574] LNetError: 149004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2659.209889] LNet: Removed LNI 192.168.206.52@tcp2 [ 2659.789726] Key type .llcrypt unregistered [ 2659.792868] Key type ._llcrypt unregistered [ 2672.959865] Key type ._llcrypt registered [ 2672.963624] Key type .llcrypt registered [ 2674.423396] Lustre: DEBUG MARKER: == sanity-lnet test 636: routed Push keeps the groups apart ========================================================== 08:51:17 (1788180677) [ 2676.424511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_636 Need at least 2 remote nodes found oleg652-server [ 2678.371577] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2457 sec ========== 08:51:21 (1788180681) [ 2679.265488] Key type .llcrypt unregistered [ 2679.267799] Key type ._llcrypt unregistered [ 2692.537976] Key type ._llcrypt registered [ 2692.540365] Key type .llcrypt registered [ 2693.193267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2693.210844] alg: No test for adler32 (adler32-zlib) [ 2694.349253] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2694.627993] LNet: Added LNI 192.168.206.52@tcp [8/256/0/180] [ 2696.303620] Key type lgssc registered [ 2697.684306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2761.913467] Lustre: Mounted lustre-client [ 2767.249087] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2774.389247] Lustre: Unmounted lustre-client [ 2800.685261] Key type lgssc unregistered [ 2800.932717] LNet: 153037:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2800.945511] LNetError: 153037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2800.968236] LNet: Removed LNI 192.168.206.52@tcp [ 2801.739143] Key type .llcrypt unregistered [ 2801.740787] Key type ._llcrypt unregistered