[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419168675 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003235] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008436] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011144] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022538] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027300] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.033398] *** VALIDATE proc *** [ 0.034249] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037227] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043666] debug: unmapping init [mem 0xffffffff99659000-0xffffffff99660fff] [ 0.045168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054343] rcu: Hierarchical SRCU implementation. [ 0.056450] smp: Bringing up secondary CPUs ... [ 0.057585] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061610] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135696] node 0 deferred pages initialised in 69ms [ 0.139058] devtmpfs: initialized [ 0.140315] x86/mm: Memory block size: 128MB [ 0.144071] gcov: version magic: 0x41383552 [ 0.147227] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152323] pinctrl core: initialized pinctrl subsystem [ 0.154199] [ 0.154770] ************************************************************* [ 0.157013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159011] ** ** [ 0.161012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163015] ** ** [ 0.165011] ** This means that this kernel is built to expose internal ** [ 0.168013] ** IOMMU data structures, which may compromise security on ** [ 0.170010] ** your system. ** [ 0.172011] ** ** [ 0.174013] ** If you see this message and you are not debugging the ** [ 0.176013] ** kernel, report this immediately to your vendor! ** [ 0.178012] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182012] ************************************************************* [ 0.184718] NET: Registered protocol family 16 [ 0.187430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195115] cpuidle: using governor menu [ 0.196645] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199432] PCI: Using configuration type 1 for base access [ 0.201144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210455] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217050] cryptd: max_cpu_qlen set to 1000 [ 0.219083] ACPI: Added _OSI(Module Device) [ 0.221015] ACPI: Added _OSI(Processor Device) [ 0.223013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230789] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240044] ACPI: Interpreter enabled [ 0.241056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242013] ACPI: Using IOAPIC for interrupt routing [ 0.244093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257337] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268263] acpiphp: Slot [2] registered [ 0.270111] acpiphp: Slot [5] registered [ 0.271096] acpiphp: Slot [6] registered [ 0.272114] acpiphp: Slot [3] registered [ 0.274087] acpiphp: Slot [4] registered [ 0.275117] acpiphp: Slot [7] registered [ 0.276104] acpiphp: Slot [8] registered [ 0.278113] acpiphp: Slot [9] registered [ 0.279092] acpiphp: Slot [10] registered [ 0.280139] acpiphp: Slot [11] registered [ 0.281084] acpiphp: Slot [12] registered [ 0.283081] acpiphp: Slot [13] registered [ 0.284100] acpiphp: Slot [14] registered [ 0.285102] acpiphp: Slot [15] registered [ 0.286083] acpiphp: Slot [16] registered [ 0.288078] acpiphp: Slot [17] registered [ 0.289090] acpiphp: Slot [18] registered [ 0.290097] acpiphp: Slot [19] registered [ 0.292105] acpiphp: Slot [20] registered [ 0.293113] acpiphp: Slot [21] registered [ 0.295099] acpiphp: Slot [22] registered [ 0.296102] acpiphp: Slot [23] registered [ 0.298097] acpiphp: Slot [24] registered [ 0.299084] acpiphp: Slot [25] registered [ 0.300078] acpiphp: Slot [26] registered [ 0.301092] acpiphp: Slot [27] registered [ 0.303084] acpiphp: Slot [28] registered [ 0.304107] acpiphp: Slot [29] registered [ 0.305107] acpiphp: Slot [30] registered [ 0.306084] acpiphp: Slot [31] registered [ 0.308059] PCI host bridge to bus 0000:00 [ 0.309020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.311020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.313022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325868] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329118] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335925] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.339991] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354493] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356720] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.361729] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365877] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.374016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385940] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421742] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.427014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.454083] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.456360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.459333] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.461351] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.464231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.470033] iommu: Default domain type: Passthrough [ 0.471342] SCSI subsystem initialized [ 0.472115] ACPI: bus type USB registered [ 0.474105] usbcore: registered new interface driver usbfs [ 0.475075] usbcore: registered new interface driver hub [ 0.477076] usbcore: registered new device driver usb [ 0.479195] pps_core: LinuxPPS API ver. 1 registered [ 0.481013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.484065] PTP clock support registered [ 0.485138] EDAC MC: Ver: 3.0.0 [ 0.487112] PCI: Using ACPI for IRQ routing [ 0.488574] NetLabel: Initializing [ 0.489009] NetLabel: domain hash size = 128 [ 0.490010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.492057] NetLabel: unlabeled traffic allowed by default [ 0.493230] vgaarb: loaded [ 0.494233] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.496011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.503101] clocksource: Switched to clocksource kvm-clock [ 0.606312] VFS: Disk quotas dquot_6.6.0 [ 0.607640] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.609936] *** VALIDATE ramfs *** [ 0.611092] *** VALIDATE hugetlbfs *** [ 0.612513] pnp: PnP ACPI init [ 0.614776] pnp: PnP ACPI: found 6 devices [ 0.634719] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.637503] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.639419] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.641160] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.643074] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.645159] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.647668] NET: Registered protocol family 2 [ 0.649736] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.654210] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.657489] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.662335] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.665673] TCP: Hash tables configured (established 65536 bind 65536) [ 0.668249] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.671167] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.673546] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.676471] NET: Registered protocol family 1 [ 0.679408] RPC: Registered named UNIX socket transport module. [ 0.681878] RPC: Registered udp transport module. [ 0.683619] RPC: Registered tcp transport module. [ 0.684780] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.687044] NET: Registered protocol family 44 [ 0.688404] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.689905] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.691358] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.692947] PCI: CLS 0 bytes, default 64 [ 0.694180] Unpacking initramfs... [ 2.010678] debug: unmapping init [mem 0xffffa031bcc64000-0xffffa031bffcffff] [ 2.014779] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.016618] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.019726] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.545613] Initialise system trusted keyrings [ 2.547501] Key type blacklist registered [ 2.549356] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.559547] zbud: loaded [ 2.563143] *** VALIDATE nfs *** [ 2.564481] *** VALIDATE nfs4 *** [ 2.566307] pstore: using deflate compression [ 2.569954] Platform Keyring initialized [ 2.675149] NET: Registered protocol family 38 [ 2.676714] Key type asymmetric registered [ 2.677837] Asymmetric key parser 'x509' registered [ 2.679289] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.681898] io scheduler mq-deadline registered [ 2.683289] io scheduler kyber registered [ 2.684452] io scheduler bfq registered [ 2.685863] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.688596] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.690823] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.692578] ACPI: Power Button [PWRF] [ 2.697155] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.701498] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.707887] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.734018] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.760474] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.765504] Non-volatile memory driver v1.3 [ 2.767060] Linux agpgart interface v0.103 [ 2.795536] virtio_blk virtio1: [vda] 146704 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.797790] vda: detected capacity change from 0 to 75112448 [ 2.819484] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.824178] vdb: detected capacity change from 0 to 1073741824 [ 2.835421] libphy: Fixed MDIO Bus: probed [ 2.842112] usbcore: registered new interface driver usbserial_generic [ 2.845570] usbserial: USB Serial support registered for generic [ 2.848847] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.851817] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.852967] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.854661] mousedev: PS/2 mouse device common for all mice [ 2.857526] rtc_cmos 00:05: RTC can wake from S4 [ 2.860087] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.863028] rtc_cmos 00:05: registered as rtc0 [ 2.865601] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.867512] intel_pstate: CPU model not supported [ 2.869310] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.870504] hid: raw HID events driver (C) Jiri Kosina [ 2.873681] usbcore: registered new interface driver usbhid [ 2.874851] usbhid: USB HID core driver [ 2.875962] drop_monitor: Initializing network drop monitor service [ 2.876070] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.877367] Initializing XFRM netlink socket [ 2.880885] NET: Registered protocol family 10 [ 2.882686] Segment Routing with IPv6 [ 2.883957] NET: Registered protocol family 17 [ 2.885657] mpls_gso: MPLS GSO support [ 2.890776] RAS: Correctable Errors collector initialized. [ 2.892013] AVX version of gcm_enc/dec engaged. [ 2.892766] AES CTR mode by8 optimization enabled [ 2.978624] sched_clock: Marking stable (2978598919, 0)->(3860091920, -881493001) [ 2.981494] registered taskstats version 1 [ 2.983229] Loading compiled-in X.509 certificates [ 2.984373] zswap: loaded using pool lzo/zbud [ 3.010424] Key type big_key registered [ 3.023078] Key type encrypted registered [ 3.024718] ima: No TPM chip found, activating TPM-bypass! [ 3.026794] ima: Allocated hash algorithm: sha1 [ 3.028427] ima: No architecture policies found [ 3.029832] evm: Initialising EVM extended attributes: [ 3.031210] evm: security.selinux [ 3.032356] evm: security.ima [ 3.033464] evm: security.capability [ 3.034719] evm: HMAC attrs: 0x1 [ 3.037115] rtc_cmos 00:05: setting system clock to 2026-09-19 01:32:09 UTC (1789781529) [ 3.043553] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 3.046353] debug: unmapping init [mem 0xffffffff99382000-0xffffffff99658fff] [ 3.055108] Write protecting the kernel read-only data: 28672k [ 3.058339] debug: unmapping init [mem 0xffffffff97a03000-0xffffffff97bfffff] [ 3.060973] debug: unmapping init [mem 0xffffffff98314000-0xffffffff983fffff] [ 3.096404] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.104815] systemd[1]: Detected virtualization kvm. [ 3.107182] systemd[1]: Detected architecture x86-64. [ 3.108483] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.132257] systemd[1]: No hostname configured. [ 3.133263] systemd[1]: Set hostname to . [ 3.134421] random: systemd: uninitialized urandom read (16 bytes read) [ 3.136532] systemd[1]: Initializing machine ID from random generator. [ 3.177533] random: ln: uninitialized urandom read (6 bytes read) [ 3.260175] random: systemd: uninitialized urandom read (16 bytes read) [ 3.261716] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.265630] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.271826] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.846175] device-mapper: uevent: version 1.0.3 [ 3.847618] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.557741] virtio_net virtio0 ens2: renamed from eth0 [ 4.564195] random: fast init done [ 4.588372] scsi host0: ata_piix [ 4.628735] scsi host1: ata_piix [ 4.630062] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.632516] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.524036] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.540386] random: crng init done [ 9.541580] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.012662] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.202936] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.439836] SELinux: Disabled at runtime. [ 11.495607] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.503958] systemd[1]: Detected virtualization kvm. [ 11.505797] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.003348] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.005978] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.012311] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.016302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.019698] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.027196] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.034679] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.064977] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.540247] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.870796] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.933568] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.011113] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.026047] EDAC sbridge: Ver: 1.1.2 [ 14.355532] Key type dns_resolver registered [ 14.644644] NFS: Registering the id_resolver key type [ 14.646509] Key type id_resolver registered [ 14.648231] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg123-client login: [ 43.873497] libcfs: loading out-of-tree module taints kernel. [ 43.914583] Key type ._llcrypt registered [ 43.916209] Key type .llcrypt registered [ 44.095894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.102828] alg: No test for adler32 (adler32-zlib) [ 45.143182] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 45.511926] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 47.151230] Key type lgssc registered [ 49.590492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.253380] hrtimer: interrupt took 3299079 ns [ 201.408120] Lustre: Mounted lustre-client - version 2.17.57_110_g5f5d390 [ 207.492465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.895643] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.785165] Lustre: lustre-OST0000-osc-ffffa03204ed4800: disconnect after 23s idle [ 228.714030] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing yml_node [ 233.630737] Lustre: DEBUG MARKER: Client: 2.17.57.110 [ 236.799497] Lustre: DEBUG MARKER: MDS: 2.17.57.110 [ 239.941845] Lustre: DEBUG MARKER: OSS: 2.17.57.110 [ 241.408294] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 18 21:36:06 EDT 2026 [ 256.772256] Lustre: Unmounted lustre-client [ 285.812247] Key type lgssc unregistered [ 286.067789] LNet: 8404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.081712] LNetError: 8404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.098599] LNet: Removed LNI 192.168.201.23@tcp [ 287.550239] Key type .llcrypt unregistered [ 287.560051] Key type ._llcrypt unregistered [ 315.597590] Key type ._llcrypt registered [ 315.600353] Key type .llcrypt registered [ 315.745577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 315.755105] alg: No test for adler32 (adler32-zlib) [ 316.687559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 316.750079] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 316.768798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 321.337071] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 321.782275] LNet: 9284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.786789] LNetError: 9284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.812784] LNet: Removed LNI 192.168.201.23@tcp [ 322.322859] Key type .llcrypt unregistered [ 322.326340] Key type ._llcrypt unregistered [ 330.026231] Key type ._llcrypt registered [ 330.030345] Key type .llcrypt registered [ 331.421594] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:37:36 (1789781856) [ 332.774878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.812641] alg: No test for adler32 (adler32-zlib) [ 333.858188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.951139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.001094] LNet: 9692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.035691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.085887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 334.096732] LNet: 9699:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 343.845422] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:37:47 (1789781867) [ 344.027694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.104341] LNet: 10273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.179370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.330021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 353.386619] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:37:58 (1789781878) [ 353.571905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.632762] LNet: 10865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.669355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.767032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 361.324888] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:38:06 (1789781886) [ 361.701129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.737742] LNet: 11451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.784899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.905251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 370.509620] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:38:15 (1789781895) [ 370.803615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.834990] LNet: 12036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.869738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.949994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 371.267425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 380.421613] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:38:24 (1789781904) [ 380.539460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.600169] LNet: 12620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.650630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.868125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 381.717224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 382.851624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.769566] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:38:37 (1789781917) [ 393.099581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.189292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.275815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 401.757064] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:38:46 (1789781926) [ 402.230779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.269077] LNet: 13808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.281920] LNet: 13808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 402.338700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.438644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 402.529982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.118073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 403.174932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.786823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 403.849609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.479874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 404.532489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 405.195755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 405.235399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.701951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 405.749712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.521724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 406.601772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.531742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 407.644780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 408.332421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 408.386379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.094362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 409.172577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.149876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.190053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.944856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 411.015871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.646479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.696556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 419.635729] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:39:04 (1789781944) [ 419.728295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.781632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.865166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 419.929044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 426.925805] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:39:12 (1789781952) [ 427.271671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.322760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.396916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 427.446866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 435.886775] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:39:20 (1789781960) [ 436.195206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.236357] LNet: 15664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.241812] LNet: 15664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 436.289245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.367528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 436.433950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.785475] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:39:29 (1789781969) [ 445.062642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.163450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.302451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 445.347343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.205818] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:39:37 (1789781977) [ 453.361520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.451106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.621667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 453.671524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 461.307898] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:39:46 (1789781986) [ 461.572263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.645216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.771425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 461.807489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 469.710265] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:39:54 (1789781994) [ 470.137142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.195296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.324322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.845396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.601222] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:40:03 (1789782003) [ 478.763126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.823650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.105674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.362775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 486.815240] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:40:11 (1789782011) [ 487.050787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.107649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.218936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.511755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.939635] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:40:19 (1789782019) [ 494.163786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.253442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.881573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.345030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.479832] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:40:27 (1789782027) [ 502.659171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.693106] LNet: 20489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.703766] LNet: 20489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 502.770775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.913051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.013148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.656608] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:40:35 (1789782035) [ 510.776153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.830575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.925879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.981931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.429245] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:40:43 (1789782043) [ 518.850344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.898836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.997246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 526.973335] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:40:51 (1789782051) [ 527.381854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.437251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.542191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 534.881058] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:41:00 (1789782060) [ 535.125268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.156386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.217926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 535.538841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 542.723528] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:41:07 (1789782067) [ 542.901593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.008480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.140595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 543.176809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 549.787835] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:41:15 (1789782075) [ 550.228473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.284598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.395774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 550.431327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 558.961879] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:41:23 (1789782083) [ 559.107216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.212443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.388857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 559.457214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 567.414905] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:41:32 (1789782092) [ 567.754927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.778802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.896284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 568.005496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 575.321430] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:41:40 (1789782100) [ 575.503917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.583222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.790149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 575.920646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 575.967126] LNetError: 25808:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 576.095599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 576.186220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 582.913395] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:41:48 (1789782108) [ 583.153189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.256346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.368558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XuS6RmkOiw [ 583.497848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XuS6RmkOiw [ 583.565673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XuS6RmkOiw [ 591.382742] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:41:56 (1789782116) [ 591.720431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.762636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.861713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 591.912522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 592.008711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 599.645195] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:42:04 (1789782124) [ 599.774668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.844091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.062666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.088396] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 601.652731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 603.349879] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 605.415783] Key type lgssc registered [ 606.636635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 610.376271] LNetError: 27566:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.23@tcp99 has no available nets [ 613.573710] Key type lgssc unregistered [ 613.892993] LNetError: 28675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.910473] LNet: Removed LNI 192.168.201.23@tcp [ 614.653818] Key type .llcrypt unregistered [ 614.658848] Key type ._llcrypt unregistered [ 623.077544] Key type ._llcrypt registered [ 623.083497] Key type .llcrypt registered [ 624.988903] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:42:29 (1789782149) [ 625.423432] Key type .llcrypt unregistered [ 625.425226] Key type ._llcrypt unregistered [ 635.335333] Key type ._llcrypt registered [ 635.337615] Key type .llcrypt registered [ 635.538992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 635.549328] alg: No test for adler32 (adler32-zlib) [ 636.630892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.682772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 636.697786] LNet: 29584:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 636.702795] LNet: 29584:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 636.707770] LNet: 29584:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 636.719146] LNet: Message forwarding will be enabled in 2s [ 639.775384] LNet: Message forwarding enabled [ 640.428826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 640.497813] LNet: Message forwarding will stop in 2s to 6s [ 643.877539] LNet: Message forwarding disabled [ 644.198569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.265654] LNet: 29674:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 644.310681] LNet: 29674:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 644.362400] LNet: 29674:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 644.426549] LNet: Message forwarding will be enabled in 2s [ 644.849327] LNet: 29687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.399782] Key type .llcrypt unregistered [ 646.408074] Key type ._llcrypt unregistered [ 665.114663] Key type ._llcrypt registered [ 665.117573] Key type .llcrypt registered [ 666.487315] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:43:11 (1789782191) [ 667.293144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.317549] alg: No test for adler32 (adler32-zlib) [ 668.279213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.352320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 675.779693] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:43:20 (1789782200) [ 675.981236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.008408] LNet: 31088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.026980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.087584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 676.147793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 676.201497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 676.258173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 676.292492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 676.349224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 676.406735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 676.444605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 676.512142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 676.556831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 676.600081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 676.646300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 676.682066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 676.720309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 676.768475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 676.813150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 676.846104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 676.889552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 676.947950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 676.970986] LNetError: 31131:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 685.001049] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:43:29 (1789782209) [ 685.166177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.219932] LNet: 31698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.281639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.439917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.175517] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:43:37 (1789782217) [ 695.148716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg123-server [ 697.185751] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:43:42 (1789782222) [ 697.612692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.653315] LNet: 32634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.675792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.741761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.779396] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 698.660754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[24-28]@tcp [ 698.695926] LNet: 32767:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 698.751439] LNetError: 32638:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 698.779931] LNetError: 32638:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.26@tcp has gone from up to down [ 706.516987] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:43:51 (1789782231) [ 709.200401] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg123-server [ 711.523677] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:43:56 (1789782236) [ 711.563729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.577603] LNet: 33697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.580553] LNetError: 33697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.616768] LNet: Removed LNI 192.168.201.23@tcp [ 712.704846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.827078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.866088] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 713.939372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[24-29/2]@tcp [ 713.971500] LNet: 33822:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 713.981696] LNet: 33822:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 714.017419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[24-29/2]@tcp [ 714.025552] LNetError: 33709:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.24@tcp is being used as a gateway but routing feature is not turned on [ 714.066581] LNetError: 33709:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 714.085346] LNetError: 33709:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.24@tcp has gone from up to down [ 714.101855] LNetError: 33709:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 722.400126] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:44:06 (1789782246) [ 722.523237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.571460] LNet: 34408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.580835] LNetError: 34408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.602358] LNet: Removed LNI 192.168.201.23@tcp [ 722.627919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.743578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 722.810615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.840915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.923610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.019102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.107965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.244967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 723.382923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.452682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.573573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.744621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.825478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.947782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 724.035557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.107739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.155535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.212592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.235627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.289395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 724.402367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.502448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.598597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.688773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.758245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.816179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 724.942981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.999484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.070249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.128748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.163714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.205608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 725.256920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.289586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.414279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 732.183192] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:44:17 (1789782257) [ 734.108457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg123-server [ 736.570615] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:44:21 (1789782261) [ 739.084343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg123-server [ 741.271528] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:44:25 (1789782265) [ 741.423201] LNet: 35812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.431035] LNet: 35812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 742.485513] Key type .llcrypt unregistered [ 742.487408] Key type ._llcrypt unregistered [ 750.714832] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.020251] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.037083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.046709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.694376] Key type ._llcrypt registered [ 753.699984] Key type .llcrypt registered [ 753.919709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.927743] alg: No test for adler32 (adler32-zlib) [ 754.961492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.140870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.204646] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 756.207890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.255688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.865992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 758.920101] LNet: Removed LNI 10.1.2.3@tcp [ 759.121303] LNet: 36535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.128539] LNetError: 36535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.141701] LNet: Removed LNI 192.168.201.23@tcp [ 759.988520] Key type .llcrypt unregistered [ 759.989962] Key type ._llcrypt unregistered [ 776.124842] Key type ._llcrypt registered [ 776.126364] Key type .llcrypt registered [ 777.835680] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:45:02 (1789782302) [ 778.588565] Key type .llcrypt unregistered [ 778.590056] Key type ._llcrypt unregistered [ 785.638148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.809754] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.820969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.835317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.749328] Key type ._llcrypt registered [ 788.754672] Key type .llcrypt registered [ 788.978848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.996364] alg: No test for adler32 (adler32-zlib) [ 789.888561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.947338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.976595] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 790.632183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 790.648495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.810202] LNet: 38027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.817122] LNetError: 38027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.821697] LNet: Removed LNI 10.1.2.3@tcp [ 792.831331] LNet: Removed LNI 192.168.201.23@tcp [ 793.419535] Key type .llcrypt unregistered [ 793.421950] Key type ._llcrypt unregistered [ 811.936623] Key type ._llcrypt registered [ 811.938079] Key type .llcrypt registered [ 813.430371] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:45:38 (1789782338) [ 814.614363] Key type .llcrypt unregistered [ 814.616221] Key type ._llcrypt unregistered [ 824.117989] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 824.614537] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.640814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.671783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 826.811799] Key type ._llcrypt registered [ 826.822019] Key type .llcrypt registered [ 827.129421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.160909] alg: No test for adler32 (adler32-zlib) [ 828.518211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.702911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 828.785511] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 832.686759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 832.741077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.493803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 835.518666] LNet: Removed LNI 10.1.2.3@tcp [ 835.536934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 835.561948] LNet: Removed LNI 10.1.2.31@tcp [ 835.573359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 835.596507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.276961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 836.298241] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 837.332274] LNet: 39938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.336730] LNetError: 39938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 837.340256] LNet: Removed LNI 10.1.2.31@tcp [ 837.350172] LNet: Removed LNI 10.1.2.3@tcp [ 837.944801] Key type .llcrypt unregistered [ 837.947759] Key type ._llcrypt unregistered [ 852.962688] Key type ._llcrypt registered [ 852.969062] Key type .llcrypt registered [ 854.321340] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:46:19 (1789782379) [ 854.811698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.826946] alg: No test for adler32 (adler32-zlib) [ 855.699201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.729976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.745211] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 856.367992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.415403] LNet: 40888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.435112] LNetError: 40888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.474951] LNet: Removed LNI 192.168.201.23@tcp [ 856.522955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.599389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.23@tcp [ 856.620936] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 857.261484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.23@tcp [ 857.295974] LNet: Removed LNI 192.168.201.23@tcp [ 857.375215] LNet: 41023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.001858] Key type .llcrypt unregistered [ 858.004868] Key type ._llcrypt unregistered [ 870.653734] Key type ._llcrypt registered [ 870.657565] Key type .llcrypt registered [ 872.272855] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:46:36 (1789782396) [ 874.958757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg123-server [ 876.866226] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:46:41 (1789782401) [ 877.638302] Key type .llcrypt unregistered [ 877.640413] Key type ._llcrypt unregistered [ 885.112710] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.286292] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.294150] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.311973] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.292464] Key type ._llcrypt registered [ 888.296413] Key type .llcrypt registered [ 888.429172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.443970] alg: No test for adler32 (adler32-zlib) [ 889.360859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.433730] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 890.263517] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 892.208459] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.218107] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.230224] LNet: Removed LNI 192.168.201.23@tcp1 [ 892.244765] LNet: Removed LNI 10.1.2.3@tcp2 [ 892.740369] Key type .llcrypt unregistered [ 892.741931] Key type ._llcrypt unregistered [ 905.032958] Key type ._llcrypt registered [ 905.034245] Key type .llcrypt registered [ 906.921884] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:47:11 (1789782431) [ 907.427785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.441584] alg: No test for adler32 (adler32-zlib) [ 908.497677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.610880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 908.663455] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 909.783525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.847565] LNet: 43824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.871382] LNetError: 43824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.892532] LNet: Removed LNI 192.168.201.23@tcp [ 909.934702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.079693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 910.126383] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 919.322707] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:47:23 (1789782443) [ 919.692935] LNet: 44416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.702291] LNetError: 44416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.722724] LNet: Removed LNI 192.168.201.23@tcp [ 920.425702] Key type .llcrypt unregistered [ 920.428155] Key type ._llcrypt unregistered [ 928.627887] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.764435] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.771263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.774025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.394768] Key type ._llcrypt registered [ 931.400361] Key type .llcrypt registered [ 931.555991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.583703] alg: No test for adler32 (adler32-zlib) [ 932.456676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.531309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.572091] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 933.204763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 933.228172] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.883893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.948989] LNet: 45156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.958902] LNetError: 45156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 935.968271] LNet: Removed LNI 10.1.2.3@tcp [ 935.988248] LNet: Removed LNI 192.168.201.23@tcp [ 936.052170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.113152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 936.155751] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 936.162776] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.858548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.882921] LNet: 45183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.891952] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.904030] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 936.919840] LNet: Removed LNI 10.1.2.3@tcp [ 936.932978] LNet: Removed LNI 192.168.201.23@tcp [ 936.964217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.035809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 937.053333] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 937.060560] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.445626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 937.462841] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 938.906640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.945272] LNet: 45345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.954948] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.965815] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 938.988162] LNet: Removed LNI 10.1.2.3@tcp [ 938.998193] LNet: Removed LNI 192.168.201.23@tcp [ 939.007870] LNet: Removed LNI 192.168.201.23@tcp2 [ 939.055312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.159464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 939.188388] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 939.194427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 939.201394] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 947.268302] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:47:52 (1789782472) [ 947.394379] LNet: 45937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.408382] LNetError: 45937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.415738] LNetError: 45937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 947.425123] LNet: Removed LNI 10.1.2.3@tcp [ 947.426990] LNet: Removed LNI 192.168.201.23@tcp [ 947.442737] LNet: Removed LNI 192.168.201.23@tcp2 [ 948.383940] Key type .llcrypt unregistered [ 948.385717] Key type ._llcrypt unregistered [ 955.228225] Key type ._llcrypt registered [ 955.230232] Key type .llcrypt registered [ 955.514748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.557087] alg: No test for adler32 (adler32-zlib) [ 956.624700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.691701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 956.715136] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 957.944654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.982556] LNet: 46335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.987909] LNetError: 46335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.034269] LNet: Removed LNI 192.168.201.23@tcp [ 958.081645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.245572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 958.294304] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 959.234691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.271120] LNet: 46365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.283596] LNetError: 46365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.317350] LNet: Removed LNI 192.168.201.23@tcp [ 959.357833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.444765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 959.485218] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 959.903952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.945960] LNet: Removed LNI 192.168.201.23@tcp [ 959.979492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.056918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 960.093111] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 960.558066] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 960.767803] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 960.798727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 960.806953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 963.202937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.235951] LNet: 46473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.245961] LNet: 46473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 963.269129] LNetError: 46473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.276149] LNetError: 46473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 963.306447] LNet: Removed LNI 192.168.201.23@tcp [ 963.342649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.407462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 963.444056] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 970.628477] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:48:15 (1789782495) [ 970.759439] LNet: 47064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.775489] LNetError: 47064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.809139] LNet: Removed LNI 192.168.201.23@tcp [ 971.401940] Key type .llcrypt unregistered [ 971.405716] Key type ._llcrypt unregistered [ 978.158952] Key type ._llcrypt registered [ 978.161331] Key type .llcrypt registered [ 978.398392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.427234] alg: No test for adler32 (adler32-zlib) [ 979.393906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.443748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.477745] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 980.446806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.467455] LNet: 47466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.472644] LNetError: 47466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.489237] LNet: Removed LNI 192.168.201.23@tcp [ 980.527578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.643596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 980.683197] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 987.245710] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:48:32 (1789782512) [ 987.512869] LNet: 48059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.518923] LNetError: 48059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.550818] LNet: Removed LNI 192.168.201.23@tcp [ 988.178776] Key type .llcrypt unregistered [ 988.182232] Key type ._llcrypt unregistered [ 995.947651] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 996.245034] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 996.254600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 996.263730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 999.140818] Key type ._llcrypt registered [ 999.143486] Key type .llcrypt registered [ 999.350536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.361923] alg: No test for adler32 (adler32-zlib) [ 1000.304111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.355278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1000.378868] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1001.451555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1001.496144] LNet: 48500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.507340] LNetError: 48500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.547242] LNet: Removed LNI 192.168.201.23@tcp [ 1001.579550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.657296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1001.693924] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1009.113240] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:48:53 (1789782533) [ 1009.211575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.252460] LNet: 49092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.258846] LNetError: 49092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.281628] LNet: Removed LNI 192.168.201.23@tcp [ 1009.308590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.428277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1015.812285] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:49:00 (1789782540) [ 1016.058599] LNet: 49680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.783591] Key type .llcrypt unregistered [ 1016.787102] Key type ._llcrypt unregistered [ 1024.569914] Key type ._llcrypt registered [ 1024.576840] Key type .llcrypt registered [ 1025.172478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.205628] alg: No test for adler32 (adler32-zlib) [ 1026.210511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.313605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1026.349460] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1027.561133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.605461] LNet: 50076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.613841] LNetError: 50076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.649190] LNet: Removed LNI 192.168.201.23@tcp [ 1027.682918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.777291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1027.826458] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1034.719268] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:49:19 (1789782559) [ 1035.290750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.326247] LNet: 50669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.338145] LNetError: 50669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.373163] LNet: Removed LNI 192.168.201.23@tcp [ 1035.393246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.460823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1043.909750] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:49:28 (1789782568) [ 1045.774808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1047.818731] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:49:32 (1789782572) [ 1049.951460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1053.363423] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:49:37 (1789782577) [ 1055.902716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1059.537055] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:49:43 (1789782583) [ 1061.349906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1064.001058] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:49:48 (1789782588) [ 1065.924525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1068.400254] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:49:52 (1789782592) [ 1068.585285] LNet: 52720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.654600] Key type .llcrypt unregistered [ 1069.658512] Key type ._llcrypt unregistered [ 1078.387652] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1078.512342] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1078.515643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1078.522354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1081.462409] Key type ._llcrypt registered [ 1081.465907] Key type .llcrypt registered [ 1081.696831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1081.711167] alg: No test for adler32 (adler32-zlib) [ 1082.668992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.755704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1082.784791] LNet: Added LNI 192.168.201.23@tcp [8/512/0/180] [ 1082.795486] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1082.798810] LNet: 53043:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1082.913855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.932360] LNet: 53062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.937412] LNetError: 53062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.946117] LNet: Removed LNI 10.1.2.3@tcp [ 1082.950474] LNetError: 53062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.960772] LNet: Removed LNI 192.168.201.23@tcp [ 1082.979479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.093541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1083.138462] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1083.237140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.263539] LNet: 53086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.269420] LNetError: 53086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.279688] LNet: Removed LNI 192.168.201.23@tcp [ 1083.303637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.365025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1083.386305] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1083.395605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.400381] LNet: 53094:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1083.477921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.512321] LNet: 53112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.523450] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.536495] LNet: Removed LNI 10.1.2.3@tcp [ 1083.538166] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.560642] LNet: Removed LNI 192.168.201.23@tcp [ 1083.586735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.655538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1083.675420] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1083.681067] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.683960] LNet: 53120:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1083.859110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.882075] LNet: 53146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.889513] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.895739] LNet: Removed LNI 10.1.2.3@tcp [ 1083.898523] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.920842] LNet: Removed LNI 192.168.201.23@tcp [ 1083.945393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.000209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.016996] LNet: Added LNI 192.168.201.23@tcp [8/256/0/182] [ 1084.024968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1084.107405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.124496] LNet: 53174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.130601] LNetError: 53174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1084.134848] LNet: Removed LNI 10.1.2.3@tcp [ 1084.136738] LNetError: 53174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.161355] LNet: Removed LNI 192.168.201.23@tcp [ 1084.188958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.254820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.291444] LNet: Added LNI 192.168.201.23@tcp [8/256/0/182] [ 1084.298870] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1084.301000] LNet: 53182:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1084.380815] LNet: Removed LNI 10.1.2.3@tcp [ 1084.395165] LNet: Removed LNI 192.168.201.23@tcp [ 1084.963963] Key type .llcrypt unregistered [ 1084.968736] Key type ._llcrypt unregistered [ 1097.206386] Key type ._llcrypt registered [ 1097.217347] Key type .llcrypt registered [ 1098.620429] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:50:23 (1789782623) [ 1100.405181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1102.796860] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:50:27 (1789782627) [ 1103.506908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1103.531561] alg: No test for adler32 (adler32-zlib) [ 1104.530151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.610717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1104.643201] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1105.552557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.23 [ 1105.587463] LNet: 54447:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1105.610711] LNet: Removed LNI 192.168.201.23@tcp [ 1111.923682] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:50:37 (1789782637) [ 1112.222677] LNet: 55017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.227946] LNetError: 55017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.247946] LNet: Removed LNI 192.168.201.23@tcp [ 1112.861293] Key type .llcrypt unregistered [ 1112.862932] Key type ._llcrypt unregistered [ 1121.101114] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1121.332540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1124.168116] Key type ._llcrypt registered [ 1124.169598] Key type .llcrypt registered [ 1124.377134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1124.390975] alg: No test for adler32 (adler32-zlib) [ 1125.904894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1132.908446] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:50:58 (1789782658) [ 1133.048536] LNet: 55926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.051043] LNetError: 55926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1133.061366] LNet: Removed LNI 10.1.2.3@tcp [ 1133.618601] Key type .llcrypt unregistered [ 1133.623210] Key type ._llcrypt unregistered [ 1140.932864] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1141.119376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1141.126504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1141.136754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1143.923154] Key type ._llcrypt registered [ 1143.924575] Key type .llcrypt registered [ 1144.164512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.182161] alg: No test for adler32 (adler32-zlib) [ 1145.167399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1152.678646] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:51:17 (1789782677) [ 1154.442415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1156.757439] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:51:21 (1789782681) [ 1156.940887] LNet: 57103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.955058] LNetError: 57103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1156.997392] LNet: Removed LNI 10.1.2.3@tcp [ 1157.805219] Key type .llcrypt unregistered [ 1157.807273] Key type ._llcrypt unregistered [ 1164.959743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1165.182033] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1165.189071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1165.191144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1167.985188] Key type ._llcrypt registered [ 1167.987230] Key type .llcrypt registered [ 1168.233262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.256373] alg: No test for adler32 (adler32-zlib) [ 1169.264080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.362308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1178.163288] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:51:43 (1789782703) [ 1178.758641] LNet: 58337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.765596] LNetError: 58337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1178.781250] LNet: Removed LNI 10.1.2.3@tcp [ 1179.404699] Key type .llcrypt unregistered [ 1179.407916] Key type ._llcrypt unregistered [ 1187.975841] Key type ._llcrypt registered [ 1187.982521] Key type .llcrypt registered [ 1188.380060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1188.419184] alg: No test for adler32 (adler32-zlib) [ 1189.541979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1189.588569] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1198.976443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1199.790370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_interrupt -r 1 [ 1199.839346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_interrupt -r 1 [ 1199.890062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1199.927675] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1200.260320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.292809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.330745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.400262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.458550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.497147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.533145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.553941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.566604] LNet: 58772:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1200.585300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.613784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.643233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.666406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.692136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.720774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.742485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.161660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_dropped -r 1 [ 1201.208636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_dropped -r 1 [ 1201.234964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1201.249747] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1201.428532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.462247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.490714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.519490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.541198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.563955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.587657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.609537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.621623] LNet: 58881:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.638243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.664803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.688546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.711613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.735267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.760996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.789435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.204134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_aborted -r 1 [ 1202.241286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_aborted -r 1 [ 1202.283682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1202.300111] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1202.507367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.538714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.563756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.592252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.623396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.652690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.679851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.709579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.725217] LNet: 58991:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.738575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.768512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.793462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.836662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.875414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.909381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.911198] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1202.946748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.436573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_no_route -r 1 [ 1203.472078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_no_route -r 1 [ 1203.524962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1203.853761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.897480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1203.951780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.998574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.059286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.128977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.196155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.242755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.286526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.348521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.429781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1204.470060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.508342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.542532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.577559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.042564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_timeout -r 1 [ 1205.093362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_timeout -r 1 [ 1205.130666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1205.148436] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1205.153741] LNet: Skipped 2 previous similar messages [ 1205.418825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.454390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.486619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.529350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.557310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.585605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.613357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.633225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.648539] LNet: 59209:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.652329] LNet: 59209:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1205.672208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.697956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.728583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.771933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.799754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.827289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.855570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1206.289661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_error -r 1 [ 1206.316684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_error -r 1 [ 1206.342182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1206.534430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.557902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.586301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.609667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.627541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.657528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.679394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.701927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.725925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.753493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.773360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1206.797472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1206.828756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1206.853959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1206.884588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1207.181429] LNet: 59360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.190979] LNetError: 59360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.205319] LNet: Removed LNI 192.168.201.23@tcp [ 1207.692495] Key type .llcrypt unregistered [ 1207.697122] Key type ._llcrypt unregistered [ 1226.286804] Key type ._llcrypt registered [ 1226.288314] Key type .llcrypt registered [ 1227.624219] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:52:32 (1789782752) [ 1228.551465] Key type .llcrypt unregistered [ 1228.552788] Key type ._llcrypt unregistered [ 1234.830782] Key type ._llcrypt registered [ 1234.833603] Key type .llcrypt registered [ 1235.132766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1235.159427] alg: No test for adler32 (adler32-zlib) [ 1236.101608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1236.131057] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1243.540796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1260.219531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.272882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1260.294531] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1262.025588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_interrupt -r 1 [ 1262.062508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_interrupt -r 1 [ 1262.101780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_interrupt -r 1 [ 1262.178412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_interrupt -r 1 [ 1262.234721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_interrupt -r 1 [ 1262.285270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_interrupt -r 1 [ 1262.339433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_interrupt -r 1 [ 1262.382301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_interrupt -r 1 [ 1262.445371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1262.464843] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1262.468989] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1262.786210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.839509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.873174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.899291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.927950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.956400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.983378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.017540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.036348] LNet: 60825:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.057425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.095620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.141129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.185684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.220210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.270930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.310074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.037535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_dropped -r 1 [ 1264.120206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_dropped -r 1 [ 1264.194230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_dropped -r 1 [ 1264.237033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_dropped -r 1 [ 1264.268929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_dropped -r 1 [ 1264.297851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_dropped -r 1 [ 1264.353316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_dropped -r 1 [ 1264.408542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_dropped -r 1 [ 1264.456354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1264.473414] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1264.477538] LNet: Skipped 2 previous similar messages [ 1264.791620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.835345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.867478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.903621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.956794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1265.013602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1265.081261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1265.136760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.154307] LNet: 60946:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.170728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.202548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.254162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.360950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.423796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.500468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.506267] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1265.525627] LNet: Skipped 3 previous similar messages [ 1265.599341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1266.653367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_aborted -r 1 [ 1266.735414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_aborted -r 1 [ 1266.809985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_aborted -r 1 [ 1266.868035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_aborted -r 1 [ 1266.978373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_aborted -r 1 [ 1267.050285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_aborted -r 1 [ 1267.142681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_aborted -r 1 [ 1267.205881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_aborted -r 1 [ 1267.288974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1267.759291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1267.800444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1267.850780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1267.887471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.955980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.023643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.100592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.173233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.201316] LNet: 61068:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.223837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.280869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.337322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.394143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.449282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.515819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.584675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1269.365264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_no_route -r 1 [ 1269.415386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_no_route -r 1 [ 1269.458605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_no_route -r 1 [ 1269.514896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_no_route -r 1 [ 1269.559422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_no_route -r 1 [ 1269.620526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_no_route -r 1 [ 1269.686112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_no_route -r 1 [ 1269.760918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_no_route -r 1 [ 1269.840994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1269.875875] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1269.881882] LNet: Skipped 3 previous similar messages [ 1270.164887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.197266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.229364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.261946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.298330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.323356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.354917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.388573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.405919] LNet: 61189:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.421523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.460493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.499619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.551394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.602575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.647965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.686400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1271.619804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_timeout -r 1 [ 1271.688905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_timeout -r 1 [ 1271.776796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_timeout -r 1 [ 1271.823845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_timeout -r 1 [ 1271.906618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_timeout -r 1 [ 1271.993836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_timeout -r 1 [ 1272.073229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_timeout -r 1 [ 1272.155665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_timeout -r 1 [ 1272.253604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1272.754782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.805241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.892665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1273.005299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.076646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1273.158804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1273.266439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.339432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.414631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.479629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.535771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.645297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.741556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.802298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.851444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1274.709977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_error -r 1 [ 1274.747506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_error -r 1 [ 1274.770709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_error -r 1 [ 1274.826689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_error -r 1 [ 1274.891045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_error -r 1 [ 1274.956752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_error -r 1 [ 1275.012049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_error -r 1 [ 1275.089543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_error -r 1 [ 1275.184546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1275.223631] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1275.236332] LNet: Skipped 5 previous similar messages [ 1275.607435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.649265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.691902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.735945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.798160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.896493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.960033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.013684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.047840] LNet: 61432:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.067443] LNet: 61432:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1276.125682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.172819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.215531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.273651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.339821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.410587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.453165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1283.174517] LNet: 61508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.188101] LNetError: 61508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.195836] LNet: Removed LNI 192.168.201.23@tcp [ 1284.263412] LNet: Removed LNI 192.168.201.23@tcp1 [ 1284.743627] Key type .llcrypt unregistered [ 1284.746132] Key type ._llcrypt unregistered [ 1307.012328] Key type ._llcrypt registered [ 1307.013645] Key type .llcrypt registered [ 1308.697957] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:53:53 (1789782833) [ 1309.733336] Key type .llcrypt unregistered [ 1309.734855] Key type ._llcrypt unregistered [ 1318.808534] Key type ._llcrypt registered [ 1318.811668] Key type .llcrypt registered [ 1319.244345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1319.282987] alg: No test for adler32 (adler32-zlib) [ 1320.212765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1320.245201] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1328.212347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1329.178513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_dropped -r 1 [ 1329.298068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_dropped -r 1 [ 1329.372333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1329.390292] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1329.732487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1329.772249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1329.831674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1329.918033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1329.998311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.059840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.125736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.185443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.224517] LNet: 62756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.284880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.348099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.408244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.488337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.574271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.711304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1330.821989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1331.950911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_error -r 1 [ 1332.004517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_error -r 1 [ 1332.080414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1332.117039] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1332.583919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.654928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.767576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.839876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.932934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.995939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.064392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.120441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.147029] LNet: 62865:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.200599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.261745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.332978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.375316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.434246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.512875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.567224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1334.626650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_timeout -r 1 [ 1334.728403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_timeout -r 1 [ 1334.815377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1334.855115] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1335.327467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.363698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.417545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.480042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.537360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.574838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.617901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.659310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.680856] LNet: 62975:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1335.712357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.749821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.826136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.954755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.049417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.094736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.192835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.799342] LNet: 63016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.813216] LNetError: 63016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.857338] LNet: Removed LNI 192.168.201.23@tcp [ 1337.713662] Key type .llcrypt unregistered [ 1337.716195] Key type ._llcrypt unregistered [ 1370.705909] Key type ._llcrypt registered [ 1370.711126] Key type .llcrypt registered [ 1372.149515] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:54:56 (1789782896) [ 1373.995405] Key type .llcrypt unregistered [ 1373.999215] Key type ._llcrypt unregistered [ 1383.072213] Key type ._llcrypt registered [ 1383.074398] Key type .llcrypt registered [ 1383.387660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1383.412246] alg: No test for adler32 (adler32-zlib) [ 1384.513407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1384.575934] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1395.800789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1408.029549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.101953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1408.144907] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1410.011887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_dropped -r 1 [ 1410.040307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e remote_dropped -r 1 [ 1410.093095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_dropped -r 1 [ 1410.134357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e remote_dropped -r 1 [ 1410.182299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e remote_dropped -r 1 [ 1410.230893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e remote_dropped -r 1 [ 1410.278802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e remote_dropped -r 1 [ 1410.308436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e remote_dropped -r 1 [ 1410.357916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1410.381891] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1410.389978] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.123@tcp [ 1410.721927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1410.763419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1410.795971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1410.819861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1410.845253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.869353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1410.902396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.950697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1410.967974] LNet: 64483:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1410.992382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1411.031937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1411.087304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1411.142269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1411.191507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.264719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1411.346774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1411.415926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.998916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_error -r 1 [ 1412.042915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e remote_error -r 1 [ 1412.101162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_error -r 1 [ 1412.177556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e remote_error -r 1 [ 1412.250481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e remote_error -r 1 [ 1412.295417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e remote_error -r 1 [ 1412.328722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e remote_error -r 1 [ 1412.382692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e remote_error -r 1 [ 1412.460558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1412.488389] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1412.491446] LNet: Skipped 2 previous similar messages [ 1412.746707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1412.784124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1412.849498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1412.922031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1412.978310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1413.044452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1413.084664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1413.127727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1413.161471] LNet: 64607:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1413.188960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1413.238447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1413.292617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1413.353457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1413.416494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1413.458987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1413.496130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1413.599557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1414.196462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_timeout -r 1 [ 1414.271237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e remote_timeout -r 1 [ 1414.317983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_timeout -r 1 [ 1414.376489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e remote_timeout -r 1 [ 1414.443492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e remote_timeout -r 1 [ 1414.505495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e remote_timeout -r 1 [ 1414.580733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e remote_timeout -r 1 [ 1414.631907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e remote_timeout -r 1 [ 1414.715043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.123@tcp [ 1414.752715] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1414.995910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1415.022424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1415.069811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1415.112271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1415.185483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1415.244607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1415.303923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1415.358803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1415.389880] LNet: 64730:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1415.410935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1415.443233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1415.470304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1415.502423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1415.545374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1415.595789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.671777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1415.698821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.037426] LNet: 64808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.041886] LNetError: 64808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.046937] LNet: Removed LNI 192.168.201.23@tcp [ 1421.059812] LNet: Removed LNI 192.168.201.23@tcp1 [ 1421.488825] Key type .llcrypt unregistered [ 1421.493782] Key type ._llcrypt unregistered [ 1440.091626] Key type ._llcrypt registered [ 1440.094492] Key type .llcrypt registered [ 1441.817662] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:56:06 (1789782966) [ 1443.633646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1446.322526] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:56:10 (1789782970) [ 1447.129279] Key type .llcrypt unregistered [ 1447.131757] Key type ._llcrypt unregistered [ 1454.645471] Key type ._llcrypt registered [ 1454.649415] Key type .llcrypt registered [ 1455.006792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1455.048219] alg: No test for adler32 (adler32-zlib) [ 1456.077795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1456.102527] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1464.597648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1465.677059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e network_timeout -r 1 [ 1465.744676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e network_timeout -r 1 [ 1465.809800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1465.871457] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1466.196075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1466.240491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1466.289450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1466.341964] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1466.359660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1466.390071] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1466.426504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1466.468093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1466.515823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1466.560814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1466.575768] LNet: 66358:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.606388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1466.658312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1466.721950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1466.751280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.791746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1466.880757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1467.260966] LNet: 66397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.270129] LNetError: 66397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.331621] LNet: Removed LNI 192.168.201.23@tcp [ 1469.046497] Key type .llcrypt unregistered [ 1469.048453] Key type ._llcrypt unregistered [ 1490.556366] Key type ._llcrypt registered [ 1490.559975] Key type .llcrypt registered [ 1490.753889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.763542] alg: No test for adler32 (adler32-zlib) [ 1491.687321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1491.709494] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1499.400353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1509.933932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.994967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1510.037759] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1511.736777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e network_timeout -r 1 [ 1511.780925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e network_timeout -r 1 [ 1511.816848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e network_timeout -r 1 [ 1511.884529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e network_timeout -r 1 [ 1511.926685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e network_timeout -r 1 [ 1511.960354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e network_timeout -r 1 [ 1512.010363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e network_timeout -r 1 [ 1512.050351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e network_timeout -r 1 [ 1512.089326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1512.110234] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1512.229475] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1512.232084] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.123@tcp [ 1512.381263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1512.401315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1512.417990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1512.437881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1512.476582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1512.514583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1512.544612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1512.581781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1512.598958] LNet: 67274:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1512.614548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1512.640266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1512.671588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.697499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.740754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1512.796850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1517.647460] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1518.184149] LNet: 67348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.191373] LNetError: 67348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.200546] LNet: Removed LNI 192.168.201.23@tcp [ 1518.211426] LNet: Removed LNI 192.168.201.23@tcp1 [ 1518.897452] Key type .llcrypt unregistered [ 1518.901687] Key type ._llcrypt unregistered [ 1540.347543] Key type ._llcrypt registered [ 1540.349606] Key type .llcrypt registered [ 1542.577780] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:57:46 (1789783066) [ 1543.163551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.212842] alg: No test for adler32 (adler32-zlib) [ 1544.258551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.338938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.374396] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1545.233137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1545.245531] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1546.107969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 1546.182662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1546.226949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1546.259945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1546.298231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1546.346648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1546.386403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.23@tcp [ 1546.408970] LNetError: 68449:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.23@tcp to recovery [ 1547.359661] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1547.360122] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1549.471417] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1549.946489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1553.567519] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1554.865405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1554.941559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1555.081841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.132219] LNet: 68567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.166454] LNetError: 68567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.188312] LNet: Removed LNI 192.168.201.23@tcp [ 1555.206274] LNet: Removed LNI 192.168.201.23@tcp1 [ 1555.276164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.380496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.436485] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1556.924587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1556.982033] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1558.700708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 1558.783698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1558.903476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1559.034648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1559.094314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1559.188251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1559.260676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1559.333570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.23@tcp [ 1559.394765] LNetError: 68839:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.23@tcp to recovery [ 1559.457864] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1561.848942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1565.603918] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1565.619481] LNet: Skipped 1 previous similar message [ 1570.394307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1570.452137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1570.527699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.889390] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:58:26 (1789783106) [ 1582.155448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.221431] LNet: 69541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.245240] LNetError: 69541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.260172] LNetError: 69541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1582.272214] LNet: Removed LNI 192.168.201.23@tcp [ 1582.306076] LNet: Removed LNI 192.168.201.23@tcp1 [ 1582.363539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.494134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.552724] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1584.278621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1584.305364] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1585.980987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 1586.087720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1586.144798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1586.200788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1586.245820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1586.293696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1586.390103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.23@tcp --health 0 [ 1586.532571] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.23@tcp [ 1586.638318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1587.553249] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1587.563645] LNet: Skipped 1 previous similar message [ 1602.638359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1602.853630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1602.918202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1602.965432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.23@tcp --health 500 [ 1603.079584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1608.795915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1608.910915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.975454] LNet: 70031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.985541] LNetError: 70031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.994414] LNetError: 70031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1609.009482] LNet: Removed LNI 192.168.201.23@tcp [ 1609.027936] LNet: Removed LNI 192.168.201.23@tcp1 [ 1609.086915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.171678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.221539] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1610.328350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.364450] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1611.549570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 1611.607517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1611.665207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1611.723421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1611.775605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1611.835394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1611.867295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1611.894653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.23@tcp --health 0 [ 1612.197765] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1612.212958] LNet: Skipped 3 previous similar messages [ 1613.360333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1622.813417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1622.846954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1622.888717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.150749] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:59:15 (1789783155) [ 1630.718675] LNet: 71069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.721826] LNetError: 71069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.724304] LNetError: 71069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1630.728060] LNet: Removed LNI 192.168.201.23@tcp [ 1630.736528] LNet: Removed LNI 192.168.201.23@tcp1 [ 1631.302926] Key type .llcrypt unregistered [ 1631.313876] Key type ._llcrypt unregistered [ 1639.232236] Key type ._llcrypt registered [ 1639.237051] Key type .llcrypt registered [ 1639.628303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1639.652965] alg: No test for adler32 (adler32-zlib) [ 1640.533882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1640.572414] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1651.940300] Lustre: DEBUG MARKER: Initial discovery [ 1652.055242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.123@tcp [ 1654.915780] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1655.058957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1655.127313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.123@tcp [ 1655.168324] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1655.208638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1655.282368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.123@tcp -r 1 -m GET -l 3 [ 1655.349636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.123@tcp -r 1 -m GET -e local_error [ 1655.386435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.123@tcp -r 1 -m PUT -l 6 [ 1656.031892] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1656.929888] Lustre: DEBUG MARKER: Force oleg123-client.virtnet to discover 192.168.201.123@tcp (in background) [ 1656.963065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.123@tcp [ 1658.603572] Lustre: DEBUG MARKER: Force oleg123-server to discover 192.168.201.23@tcp [ 1660.002697] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1661.801331] Lustre: DEBUG MARKER: Wait for 71820 [ 1663.853836] Lustre: DEBUG MARKER: Finished wait on 71820 [ 1664.090482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1664.170792] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1664.214672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1664.469296] LNet: 72279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.476289] LNetError: 72279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.499180] LNet: Removed LNI 192.168.201.23@tcp [ 1665.219572] Key type .llcrypt unregistered [ 1665.224851] Key type ._llcrypt unregistered [ 1685.189062] Key type ._llcrypt registered [ 1685.191553] Key type .llcrypt registered [ 1687.162234] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:00:11 (1789783211) [ 1687.718539] Key type .llcrypt unregistered [ 1687.725143] Key type ._llcrypt unregistered [ 1694.069889] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1694.327696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1694.334222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1694.339315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1696.949681] Key type ._llcrypt registered [ 1696.951846] Key type .llcrypt registered [ 1697.138760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1697.159953] alg: No test for adler32 (adler32-zlib) [ 1698.094447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.139879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.156956] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1698.880544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1698.903618] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1708.206328] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:00:33 (1789783233) [ 1708.343302] LNet: 74346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.349367] LNetError: 74346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1708.360896] LNet: Removed LNI 10.1.2.3@tcp [ 1708.380350] LNet: Removed LNI 192.168.201.23@tcp [ 1709.008315] Key type .llcrypt unregistered [ 1709.011250] Key type ._llcrypt unregistered [ 1716.048288] Key type ._llcrypt registered [ 1716.050324] Key type .llcrypt registered [ 1716.314827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1716.333525] alg: No test for adler32 (adler32-zlib) [ 1717.285749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.338323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.355130] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1717.966517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1717.982320] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1718.965568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.23@tcp [ 1720.035450] LNetError: Unexpected error -22 connecting to 192.168.201.23@tcp at host 192.168.201.23:988 [ 1726.130748] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:00:51 (1789783251) [ 1726.488060] LNet: 75487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.499513] LNetError: 75487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1726.508073] LNet: Removed LNI 10.1.2.3@tcp [ 1726.525821] LNet: Removed LNI 192.168.201.23@tcp [ 1727.472476] Key type .llcrypt unregistered [ 1727.475951] Key type ._llcrypt unregistered [ 1734.641715] Key type ._llcrypt registered [ 1734.643240] Key type .llcrypt registered [ 1734.914616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1734.936476] alg: No test for adler32 (adler32-zlib) [ 1735.937886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.989636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1736.009620] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1736.809579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1736.823429] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 1737.540857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.23@tcp1 --nid 192.168.201.23@tcp2 [ 1741.958991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp1 192.168.201.23@tcp2 [ 1741.982917] LNetError: 76334:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.23@tcp1 to 192.168.201.23@tcp2 - Route table may be misconfigured [ 1742.007517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp2 192.168.201.23@tcp1 [ 1749.364503] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:01:14 (1789783274) [ 1749.451507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.484406] LNet: 76899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.491258] LNetError: 76899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.497556] LNet: Removed LNI 192.168.201.23@tcp1 [ 1750.567372] LNet: Removed LNI 192.168.201.23@tcp2 [ 1750.640840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.737286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.771887] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1751.497994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1751.515086] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1752.283716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 1752.319162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.23@tcp [ 1752.359232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp1 192.168.201.23@tcp1 [ 1752.401833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.23@tcp -d 192.168.201.23@tcp -e network_timeout [ 1752.447943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -e network_timeout [ 1752.526535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -e network_timeout [ 1752.572872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -e network_timeout [ 1752.612659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1752.628535] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1752.646517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1752.730643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1752.741134] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1752.831723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1758.998734] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:01:24 (1789783284) [ 1759.378825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.408580] LNet: 77771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.414313] LNetError: 77771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.426254] LNetError: 77771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1759.438711] LNet: Removed LNI 192.168.201.23@tcp [ 1760.485910] LNet: Removed LNI 192.168.201.23@tcp1 [ 1760.550722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.681625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1761.065679] LNet: 77809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.401739] Key type .llcrypt unregistered [ 1761.404518] Key type ._llcrypt unregistered [ 1772.200675] Key type ._llcrypt registered [ 1772.205295] Key type .llcrypt registered [ 1773.653255] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:01:38 (1789783298) [ 1774.201512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1774.216246] alg: No test for adler32 (adler32-zlib) [ 1775.117826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.277765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1779.345892] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:01:44 (1789783304) [ 1779.468898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.520934] LNet: 78899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.587502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.786532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.844370] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1780.785853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1780.803293] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1781.953118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1782.005169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp1 [ 1782.090455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp1 [ 1791.189888] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:01:55 (1789783315) [ 1793.946990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg123-server [ 1796.587586] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:02:00 (1789783320) [ 1798.892249] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg123-server [ 1801.313846] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:02:05 (1789783325) [ 1803.745319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg123-server [ 1805.787554] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:02:10 (1789783330) [ 1807.914837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg123-server [ 1809.787541] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:02:14 (1789783334) [ 1812.361312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg123-server [ 1814.068274] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:02:19 (1789783339) [ 1816.451356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg123-server [ 1818.872219] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:02:23 (1789783343) [ 1820.861269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg123-server [ 1823.940931] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:02:27 (1789783347) [ 1826.637396] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg123-server [ 1830.080818] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:02:34 (1789783354) [ 1833.672679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg123-server [ 1836.276473] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:02:40 (1789783360) [ 1836.578828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1836.650888] LNet: 82783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.689125] LNetError: 82783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.716202] LNet: Removed LNI 192.168.201.23@tcp [ 1837.735765] LNet: Removed LNI 192.168.201.23@tcp1 [ 1837.808338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.898476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.935095] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1838.859734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1838.924200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1839.080306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.102501] LNet: 82933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.119445] LNetError: 82933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.126372] LNetError: 82933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1839.150503] LNet: Removed LNI 192.168.201.23@tcp [ 1839.199898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.266367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1839.292480] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1841.064777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1841.497511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.539960] LNet: 83079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.550413] LNetError: 83079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.572078] LNet: Removed LNI 192.168.201.23@tcp [ 1841.629805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.723512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1841.759973] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1842.885158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1842.979109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1843.299703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.390162] LNet: Removed LNI 192.168.201.23@tcp [ 1844.428424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.501536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1844.535549] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1845.380685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1845.619767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.653396] LNet: 83371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.659354] LNet: 83371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1845.663427] LNetError: 83371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.667636] LNetError: 83371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1846.695704] LNet: Removed LNI 192.168.201.23@tcp [ 1846.717200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.789914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.823598] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1847.537517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1847.660523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1847.943220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1848.999617] LNet: Removed LNI 192.168.201.23@tcp [ 1849.031374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.083321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1849.097751] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1850.075284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1851.604147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.631262] LNetError: 83673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.641536] LNetError: 83673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1852.717681] LNet: Removed LNI 192.168.201.23@tcp [ 1852.750922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.821924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.857278] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1853.574961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1853.649427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1853.898164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.943464] LNet: 83821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.957978] LNet: 83821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1855.015388] LNet: Removed LNI 192.168.201.23@tcp [ 1855.073394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.209155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1855.257382] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1856.554425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1856.870841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.962468] LNet: Removed LNI 192.168.201.23@tcp [ 1858.005898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.158980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1858.244793] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1859.339623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1859.418948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1859.712616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.741906] LNetError: 84114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.747618] LNetError: 84114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1859.759810] LNet: Removed LNI 192.168.201.23@tcp [ 1859.774232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.827509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1859.858115] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1860.450458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1860.767972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.866140] LNet: Removed LNI 192.168.201.23@tcp [ 1861.899252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.027291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1862.066114] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1863.190592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1863.339555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1863.764460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.806579] LNet: Removed LNI 192.168.201.23@tcp [ 1864.890373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.963809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1865.046800] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1866.122677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 1866.462286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.574202] LNet: Removed LNI 192.168.201.23@tcp [ 1867.630358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.755447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1867.786262] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1868.664770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1868.809468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.915230] LNet: Removed LNI 192.168.201.23@tcp [ 1869.008764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.131797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.189924] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1870.337323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1870.474378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.500644] LNet: 84822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.505097] LNet: 84822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1870.519778] LNet: Removed LNI 192.168.201.23@tcp [ 1870.558641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.666609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1870.791751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.879679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.001452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1871.157140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.211278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.317469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1871.443673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.530934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.610965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1871.656790] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1872.756859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.795908] LNet: Removed LNI 192.168.201.23@tcp [ 1872.837965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.907075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1881.783854] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:03:26 (1789783406) [ 1881.935952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.030928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.138431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1882.159649] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/180] [ 1883.619462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.669689] LNetError: 85713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.678945] LNetError: 85713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1883.734161] LNet: Removed LNI 192.168.201.23@tcp231 [ 1883.761459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.849947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1883.883751] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/180] [ 1883.894945] LNet: 85721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1884.644963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1884.681246] LNet: Removed LNI 192.168.201.23@tcp231 [ 1884.707220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1884.751567] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/180] [ 1886.001089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.053252] LNet: Removed LNI 192.168.201.23@tcp231 [ 1886.071755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.176275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1886.220380] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/0] [ 1886.225567] LNet: 85873:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1886.595318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1886.627320] LNet: Removed LNI 192.168.201.23@tcp231 [ 1886.646906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1886.702044] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/0] [ 1887.726897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.763206] LNet: Removed LNI 192.168.201.23@tcp231 [ 1887.798136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.894991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1887.924516] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/60] [ 1887.930115] LNet: 86025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1888.301231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1888.364283] LNet: Removed LNI 192.168.201.23@tcp231 [ 1888.398414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1888.432753] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/60] [ 1889.387608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.403166] LNet: Removed LNI 192.168.201.23@tcp231 [ 1889.414205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.460255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1889.478989] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/180] [ 1889.931974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1889.959569] LNet: Removed LNI 192.168.201.23@tcp231 [ 1889.970291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1889.993807] LNet: Added LNI 192.168.201.23@tcp231 [8/256/0/180] [ 1898.304255] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:03:43 (1789783423) [ 1898.740346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.804951] LNet: Removed LNI 192.168.201.23@tcp231 [ 1898.819890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.873711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1898.890869] LNet: Added LNI 192.168.201.23@tcp232 [8/256/0/180] [ 1899.632946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1899.676565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1899.712785] LNet: Removed LNI 192.168.201.23@tcp232 [ 1899.729850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1899.760735] LNet: Added LNI 192.168.201.23@tcp232 [8/256/0/180] [ 1900.547167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1900.577343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1900.605679] LNet: Removed LNI 192.168.201.23@tcp232 [ 1900.619222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1900.640228] LNet: Added LNI 192.168.201.23@tcp232 [8/256/0/180] [ 1901.630398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1901.676583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1901.706544] LNet: Removed LNI 192.168.201.23@tcp232 [ 1901.727316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1901.751278] LNet: Added LNI 192.168.201.23@tcp232 [8/256/0/180] [ 1902.587977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1902.668156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1902.693604] LNet: Removed LNI 192.168.201.23@tcp232 [ 1902.704836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1910.645133] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:03:55 (1789783435) [ 1911.403430] LNet: 88015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.411357] LNet: 88015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1912.521452] Key type .llcrypt unregistered [ 1912.525553] Key type ._llcrypt unregistered [ 1920.375859] Key type ._llcrypt registered [ 1920.377677] Key type .llcrypt registered [ 1920.665299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1920.689531] alg: No test for adler32 (adler32-zlib) [ 1921.776296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1921.831444] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1930.285258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 1940.904089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.965793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1940.995608] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1943.062112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_interrupt -r 2 [ 1943.133610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_interrupt -r 2 [ 1943.228965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_interrupt -r 2 [ 1943.286262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_interrupt -r 2 [ 1943.354978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_interrupt -r 2 [ 1943.435158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_interrupt -r 2 [ 1943.534165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_interrupt -r 2 [ 1943.632329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_interrupt -r 2 [ 1943.704923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.123@tcp [ 1943.750913] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1943.766475] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1944.094542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1944.129803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1944.165894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1944.199569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1944.248772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1944.327501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1944.365988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1944.408507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1944.442991] LNet: 88677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1944.460871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1944.536979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1944.587938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1944.654797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1944.723486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1944.776400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1945.464570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_dropped -r 2 [ 1945.536569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_dropped -r 2 [ 1945.599861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_dropped -r 2 [ 1945.664306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_dropped -r 2 [ 1945.733430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_dropped -r 2 [ 1945.800701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_dropped -r 2 [ 1945.836593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_dropped -r 2 [ 1945.910855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_dropped -r 2 [ 1946.000568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.123@tcp [ 1946.030880] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1946.393024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1946.449914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1946.482522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1946.513084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1946.556336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1946.587393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1946.657338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1946.697851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1946.720861] LNet: 88797:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1946.750898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1946.790193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1946.829455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1946.880768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1946.957500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1947.018590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1947.661371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_aborted -r 2 [ 1947.696337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_aborted -r 2 [ 1947.734163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_aborted -r 2 [ 1947.808662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_aborted -r 2 [ 1947.888145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_aborted -r 2 [ 1947.988218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_aborted -r 2 [ 1948.020180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_aborted -r 2 [ 1948.078725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_aborted -r 2 [ 1948.132509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.123@tcp [ 1948.149838] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1948.441747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1948.480113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1948.528136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1948.568300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1948.595132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1948.619340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1948.645875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1948.674876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1948.689958] LNet: 88917:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1948.703962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1948.731826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1948.754756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1948.781649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1948.829290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1948.865353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1949.486671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_no_route -r 2 [ 1949.530819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_no_route -r 2 [ 1949.582772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_no_route -r 2 [ 1949.631119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_no_route -r 2 [ 1949.680368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_no_route -r 2 [ 1949.743221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_no_route -r 2 [ 1949.795516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_no_route -r 2 [ 1949.876585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_no_route -r 2 [ 1949.908982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.123@tcp [ 1950.165720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1950.180874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1950.218581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1950.246834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1950.309863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1950.348467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1950.388589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1950.438135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1950.482868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1950.528918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1950.564145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1950.603136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1950.662680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1950.702442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1951.551731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e local_timeout -r 2 [ 1951.616096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e local_timeout -r 2 [ 1951.669395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e local_timeout -r 2 [ 1951.749298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e local_timeout -r 2 [ 1951.813479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e local_timeout -r 2 [ 1951.851419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e local_timeout -r 2 [ 1951.903534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e local_timeout -r 2 [ 1951.941960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e local_timeout -r 2 [ 1951.971686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.123@tcp [ 1951.991829] LNet: There was an unexpected network error while writing to 192.168.201.123: rc = -22 [ 1952.003274] LNet: Skipped 1 previous similar message [ 1952.276750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1952.305907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1952.332273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1952.360503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1952.390387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1952.413412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1952.433471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1952.477046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1952.494395] LNet: 89158:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1952.501899] LNet: 89158:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1952.535633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1952.587634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1952.619300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1952.656952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1952.698193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1952.759329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1953.383826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp -m GET -e remote_dropped -r 2 [ 1953.412877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.123@tcp1 -m GET -e remote_dropped -r 2 [ 1953.446859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp -m GET -e remote_dropped -r 2 [ 1953.488839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp -d 192.168.201.23@tcp1 -m GET -e remote_dropped -r 2 [ 1953.522716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp -m GET -e remote_dropped -r 2 [ 1953.560443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.23@tcp1 -d 192.168.201.123@tcp1 -m GET -e remote_dropped -r 2 [ 1953.596930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp -m GET -e remote_dropped -r 2 [ 1953.647114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.23@tcp1 -d 192.168.201.23@tcp1 -m GET -e remote_dropped -r 2 [ 1953.711887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.23@tcp 192.168.201.123@tcp [ 1953.745345] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.123@tcp [ 1953.961838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1953.991330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1954.037876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1954.064308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1954.102101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1954.153790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1954.190980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1954.234803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1954.270076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1954.304255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1954.340979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1954.378707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1954.411664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1954.441697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1959.923820] LNet: 89353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.932328] LNetError: 89353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.947314] LNet: Removed LNI 192.168.201.23@tcp [ 1959.963214] LNet: Removed LNI 192.168.201.23@tcp1 [ 1960.492647] Key type .llcrypt unregistered [ 1960.501077] Key type ._llcrypt unregistered [ 1979.441819] Key type ._llcrypt registered [ 1979.443831] Key type .llcrypt registered [ 1981.211173] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:05:05 (1789783505) [ 1982.204863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1982.252580] alg: No test for adler32 (adler32-zlib) [ 1983.240339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.306578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1983.335738] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1984.682762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1984.708470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.23@tcp [ 1984.747635] LNetError: 90330:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.23@tcp to recovery [ 1984.854983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.23@tcp [ 1985.022059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1992.596755] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:05:17 (1789783517) [ 1992.719572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.756846] LNet: 90914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.761766] LNetError: 90914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.776450] LNet: Removed LNI 192.168.201.23@tcp [ 1992.793898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.870165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.929083] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 1993.774138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1993.799934] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 1995.016480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.23@tcp [ 1995.039689] LNetError: 91187:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.23@tcp to recovery [ 1995.935652] LNet: 1 local NIs in recovery (showing 1): 192.168.201.23@tcp [ 1995.940425] LNet: There was an unexpected network error while writing to 192.168.201.23: rc = -22 [ 1996.352243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.23@tcp [ 2005.319181] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:05:30 (1789783530) [ 2008.203897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg123-server [ 2010.566431] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:05:35 (1789783535) [ 2013.281206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg123-server [ 2016.179598] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:05:40 (1789783540) [ 2019.341173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg123-server [ 2021.493613] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:05:46 (1789783546) [ 2021.755705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.779848] LNet: 92819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.786662] LNetError: 92819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.815428] LNet: Removed LNI 192.168.201.23@tcp [ 2022.820194] LNetError: 92819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.844803] LNet: Removed LNI 192.168.201.23@tcp1 [ 2022.894215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.953312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.971980] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2024.029540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2024.147277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2024.230211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2024.448540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2024.546663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2024.689888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2034.334257] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:05:58 (1789783558) [ 2034.695415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2034.783242] LNet: 93551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.808606] LNetError: 93551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.855538] LNet: Removed LNI 192.168.201.23@tcp [ 2034.871294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.939431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2034.985403] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2043.604066] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:06:08 (1789783568) [ 2043.762788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.800783] LNet: 94256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.805719] LNetError: 94256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.820665] LNet: Removed LNI 192.168.201.23@tcp [ 2043.850496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.923599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2043.945731] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2051.264092] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:06:16 (1789783576) [ 2052.594502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2054.454349] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:06:19 (1789783579) [ 2055.235695] LNet: 95290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.239957] LNetError: 95290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.252655] LNet: Removed LNI 192.168.201.23@tcp [ 2055.904824] Key type .llcrypt unregistered [ 2055.906918] Key type ._llcrypt unregistered [ 2062.221114] Key type ._llcrypt registered [ 2062.224224] Key type .llcrypt registered [ 2062.669818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2062.745435] alg: No test for adler32 (adler32-zlib) [ 2063.748759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2063.808456] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2072.828097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 2079.836887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.123@tcp [ 2080.460864] LNet: 95705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.477325] LNetError: 95705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.501111] LNet: Removed LNI 192.168.201.23@tcp [ 2081.220573] Key type .llcrypt unregistered [ 2081.222288] Key type ._llcrypt unregistered [ 2097.792906] Key type ._llcrypt registered [ 2097.802592] Key type .llcrypt registered [ 2099.543937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2101.865721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2103.910386] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:07:08 (1789783628) [ 2106.635673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2109.057442] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:07:13 (1789783633) [ 2111.368812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg123-server [ 2113.362380] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:07:18 (1789783638) [ 2116.065094] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg123-server [ 2118.060497] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:07:22 (1789783642) [ 2120.245726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2120.280036] alg: No test for adler32 (adler32-zlib) [ 2121.323590] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 2121.531940] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2123.159336] Key type lgssc registered [ 2124.773961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2149.618485] Key type lgssc unregistered [ 2150.079131] LNet: 98879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.097769] LNetError: 98879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.142525] LNet: Removed LNI 192.168.201.23@tcp [ 2151.387481] Key type .llcrypt unregistered [ 2151.392677] Key type ._llcrypt unregistered [ 2169.675890] Key type ._llcrypt registered [ 2169.679564] Key type .llcrypt registered [ 2171.328477] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:08:15 (1789783695) [ 2171.844316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.855459] alg: No test for adler32 (adler32-zlib) [ 2172.803896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.844831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2172.878672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2172.921490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2172.944380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2180.974030] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:08:26 (1789783706) [ 2181.489392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.536524] LNet: 100292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.579541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.668927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.703182] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2182.864442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.881451] LNet: 100436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.889273] LNetError: 100436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.900146] LNet: Removed LNI 192.168.201.23@tcp [ 2182.915463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.969361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2182.991686] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2183.730788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.750627] LNetError: 100464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.769539] LNet: Removed LNI 192.168.201.23@tcp [ 2183.790786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.882615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2183.918246] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2192.838819] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:08:37 (1789783717) [ 2193.037595] LNet: 101050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.049602] LNet: 101050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2193.068107] LNetError: 101050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.092303] LNet: Removed LNI 192.168.201.23@tcp [ 2193.989723] Key type .llcrypt unregistered [ 2193.994194] Key type ._llcrypt unregistered [ 2203.424492] Key type ._llcrypt registered [ 2203.431390] Key type .llcrypt registered [ 2203.739869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2203.782954] alg: No test for adler32 (adler32-zlib) [ 2204.782058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.935815] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2204.940339] LNet: *** cfs_fail_loc=e003, val=100*** [ 2206.056926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2206.099658] LNet: *** cfs_fail_loc=e003, val=100*** [ 2206.101233] LNet: Skipped 2 previous similar messages [ 2206.142230] Ping reply received with mapping: [ 2206.150870] found metadata reply NI[192.168.201.23@tcp] [ 2206.162813] 17 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2206.170957] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2206.177820] 5a 5a 5a 5a 5a 5a 5a 5a [ 2206.430171] LNet: 101451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.452819] LNetError: 101451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.525211] LNet: Removed LNI 192.168.201.23@tcp [ 2208.420477] Key type .llcrypt unregistered [ 2208.424780] Key type ._llcrypt unregistered [ 2222.092425] Key type ._llcrypt registered [ 2222.097968] Key type .llcrypt registered [ 2223.723562] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:09:08 (1789783748) [ 2224.604406] Key type .llcrypt unregistered [ 2224.606447] Key type ._llcrypt unregistered [ 2232.082363] Key type ._llcrypt registered [ 2232.084715] Key type .llcrypt registered [ 2232.337989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2232.358836] alg: No test for adler32 (adler32-zlib) [ 2233.478980] LNetError: 102487:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2233.856555] Key type .llcrypt unregistered [ 2233.862762] Key type ._llcrypt unregistered [ 2239.672827] Key type ._llcrypt registered [ 2239.674468] Key type .llcrypt registered [ 2241.079800] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:09:25 (1789783765) [ 2241.788528] Key type .llcrypt unregistered [ 2241.800267] Key type ._llcrypt unregistered [ 2249.675661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2249.690986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2252.397901] Key type ._llcrypt registered [ 2252.400479] Key type .llcrypt registered [ 2252.592812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2252.620534] alg: No test for adler32 (adler32-zlib) [ 2253.571131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.640026] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2263.054398] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:09:47 (1789783787) [ 2263.246707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.285532] LNet: 104261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.290258] LNetError: 104261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2263.307897] LNet: Removed LNI 10.1.2.3@tcp [ 2263.330429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.396827] Key type .llcrypt unregistered [ 2264.401401] Key type ._llcrypt unregistered [ 2271.928805] Key type ._llcrypt registered [ 2271.934792] Key type .llcrypt registered [ 2272.140115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2272.168301] alg: No test for adler32 (adler32-zlib) [ 2273.172059] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2273.305466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2273.561746] LNet: 104546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.570726] LNetError: 104546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.609176] LNet: Removed LNI 192.168.201.23@tcp [ 2274.401389] Key type .llcrypt unregistered [ 2274.403174] Key type ._llcrypt unregistered [ 2289.289879] Key type ._llcrypt registered [ 2289.292272] Key type .llcrypt registered [ 2290.849784] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:10:15 (1789783815) [ 2291.659941] Key type .llcrypt unregistered [ 2291.664896] Key type ._llcrypt unregistered [ 2299.583985] Key type ._llcrypt registered [ 2299.585824] Key type .llcrypt registered [ 2299.900685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2299.926537] alg: No test for adler32 (adler32-zlib) [ 2310.514519] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:10:35 (1789783835) [ 2310.698726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.786297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.854614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.887491] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2311.775895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.816596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2311.847593] LNet: Removed LNI 192.168.201.23@tcp [ 2312.063635] LNet: 106286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.502577] Key type .llcrypt unregistered [ 2312.506787] Key type ._llcrypt unregistered [ 2325.239650] Key type ._llcrypt registered [ 2325.244300] Key type .llcrypt registered [ 2326.618991] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:10:51 (1789783851) [ 2327.185219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2327.194963] alg: No test for adler32 (adler32-zlib) [ 2328.110956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.152632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.170190] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2328.937649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2328.964523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.23@tcp [ 2328.979398] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2328.982507] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2328.985470] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2328, 0, 0 [ 2328.989687] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2328.992531] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2328.995979] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2328.998988] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2328, 0, 0 [ 2329.002625] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2329.005327] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2329.008629] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2329.011516] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2328, 0, 2378 [ 2329.017273] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2329.019261] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2329.022598] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2329.025628] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2328, 0, 2378 [ 2329.030359] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2335.399267] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:11:00 (1789783860) [ 2335.875699] LNet: 107825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.886843] LNetError: 107825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.902636] LNet: Removed LNI 192.168.201.23@tcp [ 2336.762464] Key type .llcrypt unregistered [ 2336.770547] Key type ._llcrypt unregistered [ 2344.550935] Key type ._llcrypt registered [ 2344.553469] Key type .llcrypt registered [ 2344.715928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2344.728956] alg: No test for adler32 (adler32-zlib) [ 2345.691457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2345.763634] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2352.381634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 2361.261340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.330081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2361.377846] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 2362.506651] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2362.793763] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2362.802864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2362.818867] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2365.197444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2365.234843] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2377.838193] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:11:42 (1789783902) [ 2378.196802] LNet: 109329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.201387] LNetError: 109329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.205869] LNet: Removed LNI 192.168.201.23@tcp [ 2379.231779] LNet: Removed LNI 192.168.201.23@tcp1 [ 2379.233616] LNetError: 109329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2379.258203] LNetError: 109329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2379.291874] LNet: Removed LNI 10.1.2.3@tcp99 [ 2379.893859] Key type .llcrypt unregistered [ 2379.896350] Key type ._llcrypt unregistered [ 2386.407565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2386.411403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2389.101328] Key type ._llcrypt registered [ 2389.109221] Key type .llcrypt registered [ 2389.295037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2389.310201] alg: No test for adler32 (adler32-zlib) [ 2390.346378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.425179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.436666] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2391.062668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2391.094145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2392.729809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2392.765286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2392.798375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2392.814884] LNetError: 110070:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2392.847238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2392.881826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2393.186323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2393.241992] LNet: 110090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.246640] LNetError: 110090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2393.251176] LNet: Removed LNI 10.1.2.3@tcp [ 2394.277958] LNet: Removed LNI 192.168.201.23@tcp [ 2395.118625] Key type .llcrypt unregistered [ 2395.121283] Key type ._llcrypt unregistered [ 2409.765697] Key type ._llcrypt registered [ 2409.767741] Key type .llcrypt registered [ 2410.644938] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:12:16 (1789783936) [ 2411.353696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2411.370932] alg: No test for adler32 (adler32-zlib) [ 2412.283988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.360190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2412.399715] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2413.155451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2413.211877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg123-client@tcp [ 2419.616620] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:12:24 (1789783944) [ 2419.768414] LNet: 111617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.777703] LNetError: 111617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.826723] LNet: Removed LNI 192.168.201.23@tcp [ 2420.458282] Key type .llcrypt unregistered [ 2420.459965] Key type ._llcrypt unregistered [ 2426.366624] Key type ._llcrypt registered [ 2426.368597] Key type .llcrypt registered [ 2426.608522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2426.623841] alg: No test for adler32 (adler32-zlib) [ 2427.489251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2427.505454] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2427.527440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2434.851730] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:12:39 (1789783959) [ 2436.361627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2438.224523] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:12:43 (1789783963) [ 2440.189734] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2442.662888] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:12:47 (1789783967) [ 2442.899710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2442.977871] LNet: 113037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.003366] LNetError: 113037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.033312] LNet: Removed LNI 192.168.201.23@tcp [ 2443.084107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.204261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2443.248987] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2444.523703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2444.625569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2455.660949] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:12:59 (1789783979) [ 2455.816775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2455.866285] LNet: 113748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2455.876306] LNetError: 113748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2455.911369] LNet: Removed LNI 192.168.201.23@tcp [ 2455.941877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.039572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2456.086942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2457.075784] Key type .llcrypt unregistered [ 2457.082082] Key type ._llcrypt unregistered [ 2473.585589] Key type ._llcrypt registered [ 2473.590257] Key type .llcrypt registered [ 2475.073327] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:13:19 (1789783999) [ 2475.675155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.696865] alg: No test for adler32 (adler32-zlib) [ 2476.650455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.740282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2476.809409] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2477.933900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2477.977676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2487.529815] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:13:32 (1789784012) [ 2488.372078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.408405] LNet: 115304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.418578] LNetError: 115304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.452046] LNet: Removed LNI 192.168.201.23@tcp [ 2488.526566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2488.649790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2488.802276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2498.930371] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:13:43 (1789784023) [ 2499.007124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2499.021152] LNet: 115880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.062281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2499.181511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2499.985082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2500.013858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2500.074524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2500.651020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2500.686412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2500.757174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2500.778402] LNet: 115918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.804197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.917199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2501.333134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2501.374699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2501.445693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2501.689372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2501.721871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2508.577499] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:13:53 (1789784033) [ 2509.139293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2509.168893] LNet: 116520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.210502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.281793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2509.346283] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2510.316493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2510.332303] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2511.520608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2511.572203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2511.764737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.780879] LNetError: 116799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2512.801028] LNet: Removed LNI 192.168.201.23@tcp [ 2512.810873] LNetError: 116799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2512.843734] LNet: Removed LNI 192.168.201.23@tcp2 [ 2512.929345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.065049] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2513.079727] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2513.084473] LNet: 116805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2513.899716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2514.157474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2514.215123] LNet: 116843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.223421] LNet: 116843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2515.231098] LNet: Removed LNI 192.168.201.23@tcp [ 2515.236420] LNetError: 116843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.247225] LNetError: 116843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2515.276279] LNet: Removed LNI 192.168.201.23@tcp2 [ 2515.341894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2515.426457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2515.482773] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2516.573720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2516.616937] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2518.216912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2518.328872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2518.455806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2518.886274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2519.967327] LNet: Removed LNI 192.168.201.23@tcp [ 2520.014614] LNetError: 117127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.046223] LNetError: 117127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2520.100395] LNet: Removed LNI 192.168.201.23@tcp2 [ 2520.163931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2520.294505] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2520.311412] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2520.329489] LNet: 117133:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2521.961553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2522.449717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2522.490859] LNet: 117171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.500895] LNet: 117171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2522.521209] LNet: Removed LNI 192.168.201.23@tcp [ 2523.565313] LNet: Removed LNI 192.168.201.23@tcp2 [ 2523.698937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2523.953909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2523.999610] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2525.580206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2525.603491] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2526.574506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2526.642531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2526.871917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2527.903231] LNet: Removed LNI 192.168.201.23@tcp [ 2527.920454] LNet: Removed LNI 192.168.201.23@tcp2 [ 2527.996828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2528.127775] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2528.137478] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2528.143606] LNet: 117456:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2528.876337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2529.091040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2529.110356] LNetError: 117494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2529.115809] LNetError: 117494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2529.127059] LNet: Removed LNI 192.168.201.23@tcp [ 2529.143883] LNet: Removed LNI 192.168.201.23@tcp2 [ 2529.187489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2529.238280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2529.295457] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2530.331982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2530.353949] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2531.705508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2531.768442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2531.847817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2532.103757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2533.151171] LNet: Removed LNI 192.168.201.23@tcp [ 2534.196103] LNet: Removed LNI 192.168.201.23@tcp2 [ 2534.244840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2534.367288] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2534.380041] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2534.388796] LNet: 117784:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.805178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2535.033465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2535.078283] LNet: Removed LNI 192.168.201.23@tcp [ 2536.111360] LNet: Removed LNI 192.168.201.23@tcp2 [ 2536.215822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2536.346416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2536.409921] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2537.153760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2537.181989] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2538.124275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2538.151776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2538.210700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2538.623156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2538.642762] LNet: 118119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2538.650674] LNet: 118119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2539.679610] LNet: Removed LNI 192.168.201.23@tcp [ 2539.691569] LNet: Removed LNI 192.168.201.23@tcp2 [ 2539.717204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2539.768844] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2539.774502] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2539.779145] LNet: 118125:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2540.417489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2540.980302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2542.047631] LNet: Removed LNI 192.168.201.23@tcp [ 2542.061387] LNet: Removed LNI 192.168.201.23@tcp2 [ 2542.174317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2542.314760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2542.367908] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2543.623753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2543.688464] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2545.255612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2545.333413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2545.381980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2545.499714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2545.945804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2545.986937] LNetError: 118479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.999443] LNetError: 118479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2546.017686] LNet: Removed LNI 192.168.201.23@tcp [ 2547.052043] LNet: Removed LNI 192.168.201.23@tcp2 [ 2547.106375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2547.303820] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2547.319442] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2548.621263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2559.160541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2561.460975] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:14:46 (1789784086) [ 2561.571441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2561.607942] LNet: Removed LNI 192.168.201.23@tcp [ 2562.663603] LNet: Removed LNI 192.168.201.23@tcp2 [ 2562.740994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2563.409393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2563.431562] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2564.815975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2564.865026] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2566.490301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2566.505867] LNet: Removed LNI 10.1.2.3@tcp [ 2566.548396] LNet: Removed LNI 192.168.201.23@tcp [ 2566.590760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.738218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2566.797158] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2566.800537] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2567.469804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2567.505483] LNet: Removed LNI 192.168.201.23@tcp [ 2567.521316] LNet: Removed LNI 10.1.2.3@tcp [ 2575.991878] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:15:01 (1789784101) [ 2576.755446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2576.846726] LNet: 120119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.858943] LNet: 120119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2576.928911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2577.125756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2577.205089] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2578.193370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2578.250495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.311378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.378479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.422218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.475839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.549630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.612702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.671187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.719460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2578.778816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.23@tcp [ 2579.051367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2588.312791] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:15:13 (1789784113) [ 2588.737456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2588.764642] LNetError: 120859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.772347] LNetError: 120859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2588.800714] LNet: Removed LNI 192.168.201.23@tcp [ 2588.830606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2588.955532] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2589.160646] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2589.171962] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2589.180783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2591.613919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2591.646462] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2592.457553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2592.497968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2594.831480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2594.872676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2594.946438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.23@tcp -m PUT -e local_timeout -r 1 [ 2595.021195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2595.095646] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2595.107615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2595.360948] LNetError: 120926:0:(lib-move.c:4801:lnet_parse()) 192.168.201.23@tcp, src 192.168.201.23@tcp: Dropping PUT (error -108 looking up sender) [ 2596.389509] LNet: Removed LNI 10.1.2.3@tcp [ 2597.414057] LNet: Removed LNI 192.168.201.23@tcp [ 2597.867422] Key type .llcrypt unregistered [ 2597.869386] Key type ._llcrypt unregistered [ 2620.054485] Key type ._llcrypt registered [ 2620.056810] Key type .llcrypt registered [ 2622.832178] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:15:46 (1789784146) [ 2623.763585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2623.777336] alg: No test for adler32 (adler32-zlib) [ 2624.843960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2625.479878] LNet: 122236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.651478] Key type .llcrypt unregistered [ 2626.655681] Key type ._llcrypt unregistered [ 2635.736836] Key type ._llcrypt registered [ 2635.740589] Key type .llcrypt registered [ 2636.190498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2636.200242] alg: No test for adler32 (adler32-zlib) [ 2637.194961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2637.252250] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2646.625889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 2646.971681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2647.446126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.123@tcp [ 2647.519223] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.123@tcp [ 2647.644247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2647.676507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2647.720961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2647.774379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2647.821103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2647.856635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2647.877437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2647.895415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2647.904624] LNet: 122683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2647.923299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2647.954855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2647.979930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2648.004724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2648.032472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2648.062413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2648.655489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2648.930815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2648.990327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2649.047733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2649.109875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2649.165324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2649.203124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2649.253245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2649.308252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2649.355771] LNet: 122801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2649.399510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2649.444960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2649.492938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2649.561840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2649.599418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2649.633631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2650.473510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.123@tcp [ 2650.808513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2650.847730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2650.893203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2650.919936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2650.960554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2651.005805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2651.070876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2651.130895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2651.151628] LNet: 122918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2651.182898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2651.227552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2651.299060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2651.391029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2651.469360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2651.537689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2652.204315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2652.427369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2652.448583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2652.471471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2652.491374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2652.515382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2652.540565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2652.572591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2652.600966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2652.632956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2652.662143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2652.693394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2652.726620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2652.759609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2652.781912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2653.081889] LNet: 123081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2653.089314] LNetError: 123081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2653.105359] LNet: Removed LNI 192.168.201.23@tcp [ 2653.418716] Key type .llcrypt unregistered [ 2653.420937] Key type ._llcrypt unregistered [ 2678.443721] Key type ._llcrypt registered [ 2678.445868] Key type .llcrypt registered [ 2680.362752] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:16:44 (1789784204) [ 2681.155309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2681.165800] alg: No test for adler32 (adler32-zlib) [ 2682.215157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2683.014769] LNet: 123967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.393970] Key type .llcrypt unregistered [ 2684.403831] Key type ._llcrypt unregistered [ 2694.155915] Key type ._llcrypt registered [ 2694.159800] Key type .llcrypt registered [ 2694.378647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2694.390225] alg: No test for adler32 (adler32-zlib) [ 2695.282220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2695.306727] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2706.681968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.123@tcp [ 2716.967591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2717.053654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2717.082445] LNet: Added LNI 192.168.201.23@tcp1 [8/256/0/180] [ 2718.248954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2718.768399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.123@tcp [ 2718.882140] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.123@tcp [ 2718.995788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2719.031992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2719.063559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2719.100754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2719.128512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2719.182679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2719.208243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2719.235516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2719.257764] LNet: 124617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2719.285793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2719.337582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2719.375297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2719.411420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2719.441742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2719.469544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2720.008342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2720.208230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2720.234040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2720.256543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2720.284781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2720.311695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2720.333279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2720.354885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2720.382328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2720.396327] LNet: 124734:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2720.417277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2720.473078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2720.525300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2720.561231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2720.612589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2720.647666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2721.316698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.123@tcp [ 2721.613428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2721.642228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2721.671685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2721.701577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2721.751462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2721.805039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2721.843509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2721.895392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2721.922650] LNet: 124852:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2721.944961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2721.995737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2722.066617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2722.113600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2722.165021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2722.207109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2722.795173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2723.015608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2723.039723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2723.067559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2723.096193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2723.128378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2723.159606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2723.193766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2723.223516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2723.256241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2723.287391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2723.317542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2723.354283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2723.392942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2723.432334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2728.579798] LNet: 125050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.587619] LNetError: 125050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2728.596380] LNet: Removed LNI 192.168.201.23@tcp [ 2728.608261] LNet: Removed LNI 192.168.201.23@tcp1 [ 2729.010873] Key type .llcrypt unregistered [ 2729.016278] Key type ._llcrypt unregistered [ 2746.126697] Key type ._llcrypt registered [ 2746.131279] Key type .llcrypt registered [ 2747.766440] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:17:52 (1789784272) [ 2748.225722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2748.240460] alg: No test for adler32 (adler32-zlib) [ 2749.168879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2749.291256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2749.356682] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2750.484391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2750.501678] LNet: Added LNI 192.168.201.23@tcp2 [8/256/0/180] [ 2751.134808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.23@tcp [ 2751.203358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2752.223159] LNet: Removed LNI 192.168.201.23@tcp [ 2760.899493] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:18:05 (1789784285) [ 2762.873051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2764.815491] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:18:09 (1789784289) [ 2774.173821] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:18:18 (1789784298) [ 2783.496995] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:18:28 (1789784308) [ 2791.475962] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:18:36 (1789784316) [ 2799.935724] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:18:44 (1789784324) [ 2811.154472] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:18:55 (1789784335) [ 2825.276067] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:19:09 (1789784349) [ 2836.788910] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:19:21 (1789784361) [ 2845.314339] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:19:30 (1789784370) [ 2856.803417] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:19:40 (1789784380) [ 2868.173867] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:19:52 (1789784392) [ 2878.193048] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:20:03 (1789784403) [ 2888.667241] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:20:12 (1789784412) [ 2897.050064] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:20:21 (1789784421) [ 2906.713331] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:20:31 (1789784431) [ 2918.041957] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:20:42 (1789784442) [ 2929.049405] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:20:53 (1789784453) [ 2937.782805] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:21:02 (1789784462) [ 2938.037103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2938.106457] LNet: 136752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.116345] LNetError: 136752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2939.173719] LNet: Removed LNI 192.168.201.23@tcp2 [ 2939.243933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2939.569195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2939.661186] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2941.519343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2941.571908] LNet: 136895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2941.584853] LNetError: 136895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2941.609808] LNet: Removed LNI 192.168.201.23@tcp [ 2941.659321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2941.754190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2941.786670] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 2951.402582] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:21:15 (1789784475) [ 2960.347631] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:21:25 (1789784485) [ 2971.389645] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:21:35 (1789784495) [ 2981.157638] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:21:46 (1789784506) [ 2989.688925] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:21:54 (1789784514) [ 2998.071491] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:22:02 (1789784522) [ 3007.313805] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:22:11 (1789784531) [ 3015.204482] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:22:20 (1789784540) [ 3023.898760] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:22:28 (1789784548) [ 3032.096402] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:22:36 (1789784556) [ 3041.069643] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:22:45 (1789784565) [ 3052.889726] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:22:57 (1789784577) [ 3067.752662] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2823 sec ========== 22:23:11 (1789784591) [ 3068.480816] LNet: 144366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3068.489283] LNetError: 144366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3068.542252] LNet: Removed LNI 192.168.201.23@tcp [ 3069.515100] Key type .llcrypt unregistered [ 3069.516807] Key type ._llcrypt unregistered [ 3083.793061] Key type ._llcrypt registered [ 3083.799965] Key type .llcrypt registered [ 3084.468659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3084.496780] alg: No test for adler32 (adler32-zlib) [ 3085.585144] Lustre: Lustre: Build Version: 2.17.57_110_g5f5d390 [ 3085.837743] LNet: Added LNI 192.168.201.23@tcp [8/256/0/180] [ 3087.586325] Key type lgssc registered [ 3089.029663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3169.446737] Lustre: Mounted lustre-client - version 2.17.57_110_g5f5d390 [ 3176.871411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3188.154460] Lustre: Unmounted lustre-client [ 3218.356249] Key type lgssc unregistered [ 3218.754068] LNet: 147330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.767927] LNetError: 147330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3218.798785] LNet: Removed LNI 192.168.201.23@tcp [ 3219.897209] Key type .llcrypt unregistered [ 3219.901733] Key type ._llcrypt unregistered