[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466387173 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002319] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007403] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.011102] LSM: Security Framework initializing [ 0.012047] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.021788] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025585] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.029435] *** VALIDATE proc *** [ 0.030229] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039858] debug: unmapping init [mem 0xffffffff9da59000-0xffffffff9da60fff] [ 0.041880] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042648] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043024] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045009] ... generic registers: 4 [ 0.046009] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048012] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050270] rcu: Hierarchical SRCU implementation. [ 0.052480] smp: Bringing up secondary CPUs ... [ 0.053507] x86: Booting SMP configuration: [ 0.054024] .... node #0, CPUs: #1 #2 #3 [ 0.058013] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.186093] node 0 deferred pages initialised in 122ms [ 0.190106] devtmpfs: initialized [ 0.191266] x86/mm: Memory block size: 128MB [ 0.193811] gcov: version magic: 0x41383552 [ 0.197105] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.198077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.199259] pinctrl core: initialized pinctrl subsystem [ 0.200169] [ 0.200501] ************************************************************* [ 0.201012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202011] ** ** [ 0.203024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.204010] ** ** [ 0.205011] ** This means that this kernel is built to expose internal ** [ 0.206012] ** IOMMU data structures, which may compromise security on ** [ 0.207013] ** your system. ** [ 0.208014] ** ** [ 0.209015] ** If you see this message and you are not debugging the ** [ 0.210012] ** kernel, report this immediately to your vendor! ** [ 0.211010] ** ** [ 0.212068] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.213011] ************************************************************* [ 0.214724] NET: Registered protocol family 16 [ 0.215418] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.217052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218446] cpuidle: using governor menu [ 0.220794] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222387] PCI: Using configuration type 1 for base access [ 0.224132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233141] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.234033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235268] cryptd: max_cpu_qlen set to 1000 [ 0.236727] ACPI: Added _OSI(Module Device) [ 0.237000] ACPI: Added _OSI(Processor Device) [ 0.238014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244636] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250305] ACPI: Interpreter enabled [ 0.251056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252010] ACPI: Using IOAPIC for interrupt routing [ 0.253096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269363] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270105] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274054] acpiphp: Slot [2] registered [ 0.274937] acpiphp: Slot [5] registered [ 0.275134] acpiphp: Slot [6] registered [ 0.276121] acpiphp: Slot [3] registered [ 0.277126] acpiphp: Slot [4] registered [ 0.278106] acpiphp: Slot [7] registered [ 0.279138] acpiphp: Slot [8] registered [ 0.280128] acpiphp: Slot [9] registered [ 0.281098] acpiphp: Slot [10] registered [ 0.282114] acpiphp: Slot [11] registered [ 0.283095] acpiphp: Slot [12] registered [ 0.284063] acpiphp: Slot [13] registered [ 0.285066] acpiphp: Slot [14] registered [ 0.286044] acpiphp: Slot [15] registered [ 0.287069] acpiphp: Slot [16] registered [ 0.288077] acpiphp: Slot [17] registered [ 0.289042] acpiphp: Slot [18] registered [ 0.289983] acpiphp: Slot [19] registered [ 0.290071] acpiphp: Slot [20] registered [ 0.290933] acpiphp: Slot [21] registered [ 0.291055] acpiphp: Slot [22] registered [ 0.292015] acpiphp: Slot [23] registered [ 0.292984] acpiphp: Slot [24] registered [ 0.293054] acpiphp: Slot [25] registered [ 0.293895] acpiphp: Slot [26] registered [ 0.294084] acpiphp: Slot [27] registered [ 0.294803] acpiphp: Slot [28] registered [ 0.295051] acpiphp: Slot [29] registered [ 0.295974] acpiphp: Slot [30] registered [ 0.296054] acpiphp: Slot [31] registered [ 0.296933] PCI host bridge to bus 0000:00 [ 0.297015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304833] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306187] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310753] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.312661] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.315070] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316485] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.317813] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.318236] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.319807] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.321779] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.326952] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332703] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.335015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.337017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.343015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.347748] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.349755] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.351012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.356013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.360677] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.361224] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.362303] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.363296] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.364148] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.367111] iommu: Default domain type: Passthrough [ 0.368334] SCSI subsystem initialized [ 0.370099] ACPI: bus type USB registered [ 0.371021] usbcore: registered new interface driver usbfs [ 0.372063] usbcore: registered new interface driver hub [ 0.373112] usbcore: registered new device driver usb [ 0.374118] pps_core: LinuxPPS API ver. 1 registered [ 0.375007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.377050] PTP clock support registered [ 0.378077] EDAC MC: Ver: 3.0.0 [ 0.379109] PCI: Using ACPI for IRQ routing [ 0.381534] NetLabel: Initializing [ 0.383016] NetLabel: domain hash size = 128 [ 0.385010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.388093] NetLabel: unlabeled traffic allowed by default [ 0.391067] vgaarb: loaded [ 0.393219] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.394007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.401000] clocksource: Switched to clocksource kvm-clock [ 0.496622] VFS: Disk quotas dquot_6.6.0 [ 0.498517] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.500492] *** VALIDATE ramfs *** [ 0.501611] *** VALIDATE hugetlbfs *** [ 0.503864] pnp: PnP ACPI init [ 0.506464] pnp: PnP ACPI: found 6 devices [ 0.522216] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.525477] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.527786] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.529951] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.532389] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.534814] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.537689] NET: Registered protocol family 2 [ 0.540267] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.545704] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.550174] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.555725] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.559244] TCP: Hash tables configured (established 65536 bind 65536) [ 0.562220] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.565429] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.568553] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.571761] NET: Registered protocol family 1 [ 0.574421] RPC: Registered named UNIX socket transport module. [ 0.577736] RPC: Registered udp transport module. [ 0.579696] RPC: Registered tcp transport module. [ 0.581491] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.584174] NET: Registered protocol family 44 [ 0.585502] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.587783] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.590146] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.592562] PCI: CLS 0 bytes, default 64 [ 0.594363] Unpacking initramfs... [ 1.990881] debug: unmapping init [mem 0xffff88b67cc64000-0xffff88b67ffcffff] [ 1.994441] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.996135] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.998288] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.489693] Initialise system trusted keyrings [ 2.491085] Key type blacklist registered [ 2.492619] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.501553] zbud: loaded [ 2.504496] *** VALIDATE nfs *** [ 2.505772] *** VALIDATE nfs4 *** [ 2.507437] pstore: using deflate compression [ 2.511089] Platform Keyring initialized [ 2.605862] NET: Registered protocol family 38 [ 2.607502] Key type asymmetric registered [ 2.608901] Asymmetric key parser 'x509' registered [ 2.611245] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.615743] io scheduler mq-deadline registered [ 2.617528] io scheduler kyber registered [ 2.619368] io scheduler bfq registered [ 2.624410] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.627469] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.630428] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.633280] ACPI: Power Button [PWRF] [ 2.639799] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.648731] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.662216] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.689515] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.718525] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.724427] Non-volatile memory driver v1.3 [ 2.726414] Linux agpgart interface v0.103 [ 2.762161] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.766377] vda: detected capacity change from 0 to 74829824 [ 2.791706] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.795560] vdb: detected capacity change from 0 to 1073741824 [ 2.813233] libphy: Fixed MDIO Bus: probed [ 2.822815] usbcore: registered new interface driver usbserial_generic [ 2.825434] usbserial: USB Serial support registered for generic [ 2.827405] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.831710] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.833418] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.835586] mousedev: PS/2 mouse device common for all mice [ 2.838568] rtc_cmos 00:05: RTC can wake from S4 [ 2.841142] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.842775] rtc_cmos 00:05: registered as rtc0 [ 2.847811] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.850898] intel_pstate: CPU model not supported [ 2.854645] hid: raw HID events driver (C) Jiri Kosina [ 2.855539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.858871] usbcore: registered new interface driver usbhid [ 2.858877] usbhid: USB HID core driver [ 2.860066] drop_monitor: Initializing network drop monitor service [ 2.873838] Initializing XFRM netlink socket [ 2.873923] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.875960] NET: Registered protocol family 10 [ 2.949411] Segment Routing with IPv6 [ 2.950477] NET: Registered protocol family 17 [ 2.952199] mpls_gso: MPLS GSO support [ 2.956177] RAS: Correctable Errors collector initialized. [ 2.957518] AVX version of gcm_enc/dec engaged. [ 2.958585] AES CTR mode by8 optimization enabled [ 3.038451] sched_clock: Marking stable (3038419843, 0)->(3967306623, -928886780) [ 3.042986] registered taskstats version 1 [ 3.045223] Loading compiled-in X.509 certificates [ 3.047461] zswap: loaded using pool lzo/zbud [ 3.070913] Key type big_key registered [ 3.085123] Key type encrypted registered [ 3.086189] ima: No TPM chip found, activating TPM-bypass! [ 3.087753] ima: Allocated hash algorithm: sha1 [ 3.089417] ima: No architecture policies found [ 3.092127] evm: Initialising EVM extended attributes: [ 3.093972] evm: security.selinux [ 3.095229] evm: security.ima [ 3.096278] evm: security.capability [ 3.097557] evm: HMAC attrs: 0x1 [ 3.100521] rtc_cmos 00:05: setting system clock to 2026-08-27 08:02:06 UTC (1787817726) [ 3.106945] debug: unmapping init [mem 0xffffffff9ea03000-0xffffffff9ebfffff] [ 3.110402] debug: unmapping init [mem 0xffffffff9d782000-0xffffffff9da58fff] [ 3.118071] Write protecting the kernel read-only data: 28672k [ 3.121830] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.124268] debug: unmapping init [mem 0xffffffff9c714000-0xffffffff9c7fffff] [ 3.156457] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.165439] systemd[1]: Detected virtualization kvm. [ 3.167440] systemd[1]: Detected architecture x86-64. [ 3.169307] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.193358] systemd[1]: No hostname configured. [ 3.195160] systemd[1]: Set hostname to . [ 3.197282] random: systemd: uninitialized urandom read (16 bytes read) [ 3.199844] systemd[1]: Initializing machine ID from random generator. [ 3.259631] random: ln: uninitialized urandom read (6 bytes read) [ 3.359285] random: systemd: uninitialized urandom read (16 bytes read) [ 3.362946] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.367983] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.372817] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.054238] device-mapper: uevent: version 1.0.3 [ 4.056530] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.928350] virtio_net virtio0 ens2: renamed from eth0 [ 4.958911] scsi host0: ata_piix [ 4.969473] scsi host1: ata_piix [ 4.971148] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.974141] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.343379] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.626064] random: crng init done [ 9.630314] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.112240] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.598510] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.526461] SELinux: Disabled at runtime. [ 16.635731] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.653108] systemd[1]: Detected virtualization kvm. [ 16.654708] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.467807] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.471954] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.479616] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.490558] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.495231] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.509102] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.524824] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 18.573234] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.034309] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.276459] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.426763] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.897376] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.009370] EDAC sbridge: Ver: 1.1.2 [ 25.521901] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.357541] NFS: Registering the id_resolver key type [ 26.362478] Key type id_resolver registered [ 26.367155] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-client login: [ 94.790803] libcfs: loading out-of-tree module taints kernel. [ 94.914130] Key type ._llcrypt registered [ 94.916921] Key type .llcrypt registered [ 95.278706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.284470] alg: No test for adler32 (adler32-zlib) [ 97.236145] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 98.291398] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 100.143213] Key type lgssc registered [ 102.129427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.226890] hrtimer: interrupt took 7015309 ns [ 246.049915] Lustre: Mounted lustre-client - version 2.17.57_80_gdb9f866 [ 251.961216] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 264.912720] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing check_logdir /tmp/testlogs/ [ 271.042219] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing yml_node [ 271.845965] Lustre: lustre-OST0000-osc-ffff88b6e0cde800: disconnect after 23s idle [ 276.159498] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 279.873532] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 282.930789] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 285.091407] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 04:06:46 EDT 2026 [ 300.700512] Lustre: Unmounted lustre-client [ 325.585617] Key type lgssc unregistered [ 325.878708] LNet: 8408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.885498] LNetError: 8408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.908398] LNet: Removed LNI 192.168.202.10@tcp [ 326.798683] Key type .llcrypt unregistered [ 326.801030] Key type ._llcrypt unregistered [ 351.235903] Key type ._llcrypt registered [ 351.237204] Key type .llcrypt registered [ 351.464882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 351.501753] alg: No test for adler32 (adler32-zlib) [ 352.444460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 352.517823] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 352.552432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 356.745958] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 356.971236] LNet: 9292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.980784] LNetError: 9292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.022256] LNet: Removed LNI 192.168.202.10@tcp [ 357.594645] Key type .llcrypt unregistered [ 357.595779] Key type ._llcrypt unregistered [ 363.579173] Key type ._llcrypt registered [ 363.585554] Key type .llcrypt registered [ 364.837928] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:08:06 (1787818086) [ 365.250051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 365.261704] alg: No test for adler32 (adler32-zlib) [ 366.292503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.488798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.513773] LNet: 9714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.540060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.602850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.638423] LNet: 9721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.539981] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:08:15 (1787818095) [ 373.715711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.759924] LNet: 10297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.818285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.905893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 380.443909] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:08:22 (1787818102) [ 380.680639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.716437] LNet: 10885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.746161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.834567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 388.014532] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:08:30 (1787818110) [ 388.240397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.273031] LNet: 11469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.334594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.430271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 396.164333] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:08:37 (1787818117) [ 396.414371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.478063] LNet: 12055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.526402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.627559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 397.204900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 404.082420] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:08:45 (1787818125) [ 404.158068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.203094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.281788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 404.930975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 405.336970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.760921] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:08:53 (1787818133) [ 410.835940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.867523] LNet: 13233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.878357] LNet: 13233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 410.914715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.996635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 417.467278] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:08:59 (1787818139) [ 417.730681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.775413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.841927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 417.878966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.217455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 418.251241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 418.550187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 418.575061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.821988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 418.840725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 419.188864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 419.265348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 420.025656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 420.088306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.862744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 420.949262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 421.738611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 421.772695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.388176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 422.469620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 423.239755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 423.328553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.903734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 423.962566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 424.793871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 424.889209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 426.001050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 426.088620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 435.037985] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:09:16 (1787818156) [ 435.232613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.327563] LNet: 14506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.330495] LNet: 14506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 435.397811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.466464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 435.494761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 441.204815] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:09:23 (1787818163) [ 441.314826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.406829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.526853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 441.570623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 447.539384] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:09:29 (1787818169) [ 447.901434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.951839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.079270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 448.148962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 457.073218] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:09:38 (1787818178) [ 457.226457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.312752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.468230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 457.529937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 468.388205] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:09:49 (1787818189) [ 468.552116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.579941] LNet: 16848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.605914] LNet: 16848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 468.637203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.756729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 468.811880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 477.585768] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:09:59 (1787818199) [ 477.650608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.699655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.786986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 477.816625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.932763] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:10:07 (1787818207) [ 486.115414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.241340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.397766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.712610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.559428] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:10:16 (1787818216) [ 494.681164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.771092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.087133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.309577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.461261] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:10:25 (1787818225) [ 503.590787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.615593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.768156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.088767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 512.942309] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:10:34 (1787818234) [ 513.081585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.125366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.981922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 514.540764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.332867] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:10:44 (1787818244) [ 522.544235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.664198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.777348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 522.816272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.280344] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:10:51 (1787818251) [ 529.413196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.465925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.561525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 529.680780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 538.893447] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:10:59 (1787818259) [ 539.080437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.158283] LNet: 21706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.164312] LNet: 21706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 539.241678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.498579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 549.358212] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:11:10 (1787818270) [ 549.510976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.578639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.742851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 557.324418] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:11:19 (1787818279) [ 557.403657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.451550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.553665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 557.866479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 563.278280] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:11:25 (1787818285) [ 563.542798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.595408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.662892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 563.695449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 570.961853] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:11:32 (1787818292) [ 571.075321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.133664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.228508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 571.273111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 578.803652] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:11:40 (1787818300) [ 578.926922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.992524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.100719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 579.132785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 587.386453] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:11:49 (1787818309) [ 587.627088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.673868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.772877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 587.823278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 594.031606] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:11:56 (1787818316) [ 594.187591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.232050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.277728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 594.339731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 594.352253] LNetError: 25830:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 594.368646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 594.392767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 600.471996] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:12:02 (1787818322) [ 600.723322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.749966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.798856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wpjzWa06al [ 600.861587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wpjzWa06al [ 600.894504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wpjzWa06al [ 605.676068] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:12:07 (1787818327) [ 605.866647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.902510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.957700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 605.998470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 606.043644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 611.633517] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 04:12:13 (1787818333) [ 611.961583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.037985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.146357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 612.187149] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 613.692523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 615.069143] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 617.009423] Key type lgssc registered [ 618.478008] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.523924] LNetError: 27590:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.10@tcp99 has no available nets [ 624.664493] Key type lgssc unregistered [ 625.184760] LNetError: 28701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.210412] LNet: Removed LNI 192.168.202.10@tcp [ 625.974256] Key type .llcrypt unregistered [ 625.981893] Key type ._llcrypt unregistered [ 632.874092] Key type ._llcrypt registered [ 632.878798] Key type .llcrypt registered [ 634.259943] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:12:36 (1787818356) [ 634.769319] Key type .llcrypt unregistered [ 634.770694] Key type ._llcrypt unregistered [ 642.559928] Key type ._llcrypt registered [ 642.561930] Key type .llcrypt registered [ 642.699353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 642.717982] alg: No test for adler32 (adler32-zlib) [ 643.721519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.883451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 643.910660] LNet: 29622:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 643.925416] LNet: 29622:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 643.946585] LNet: 29622:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 644.001105] LNet: Message forwarding will be enabled in 2s [ 647.135321] LNet: Message forwarding enabled [ 647.596620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 647.626466] LNet: Message forwarding will stop in 2s to 6s [ 651.233517] LNet: Message forwarding disabled [ 652.228729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 652.299514] LNet: 29721:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 652.311469] LNet: 29721:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 652.319442] LNet: 29721:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 652.349379] LNet: Message forwarding will be enabled in 2s [ 652.554983] LNet: 29734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.195447] Key type .llcrypt unregistered [ 653.201374] Key type ._llcrypt unregistered [ 667.486382] Key type ._llcrypt registered [ 667.493661] Key type .llcrypt registered [ 669.906372] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:13:10 (1787818390) [ 670.712928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.746021] alg: No test for adler32 (adler32-zlib) [ 672.075917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.304467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 679.309077] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:13:21 (1787818401) [ 679.596719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.637893] LNet: 31143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.669893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.735991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 679.780910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 679.835346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 679.876558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 679.907262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 679.935678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 680.000814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 680.057827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 680.106961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 680.163988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 680.213829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 680.258416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 680.300898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 680.344721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 680.409548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 680.481961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 680.521825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 680.567933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 680.592602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 680.612131] LNetError: 31187:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 686.698257] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:13:28 (1787818408) [ 686.827358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.858523] LNet: 31752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.893638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.989338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.138490] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:13:36 (1787818416) [ 696.153285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg210-server [ 697.745672] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:13:39 (1787818419) [ 697.837949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.871913] LNet: 32683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.907709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.008289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.039672] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 698.804398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[11-15]@tcp [ 698.827442] LNet: 32823:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 698.865396] LNetError: 32692:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 698.895599] LNetError: 32692:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.11@tcp has gone from up to down [ 705.233368] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:13:47 (1787818427) [ 706.749793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg210-server [ 708.292365] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:13:50 (1787818430) [ 708.354495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.393410] LNet: 33752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.405203] LNetError: 33752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.427298] LNet: Removed LNI 192.168.202.10@tcp [ 708.472262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.534862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.566121] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 709.269936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 709.290692] LNet: 33875:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 709.299261] LNet: 33875:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 709.314268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 709.318139] LNetError: 33767:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.15@tcp has gone from up to down [ 709.340224] LNetError: 33767:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 716.208612] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:13:58 (1787818438) [ 716.357784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.412105] LNet: 34460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.419647] LNetError: 34460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.460479] LNet: Removed LNI 192.168.202.10@tcp [ 716.502677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.657827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 716.723380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.756729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.820782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 716.886516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.938434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.030107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 717.124593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.181635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.279953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.367729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.419509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.484700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 717.613745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.656062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.739528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.825635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.880722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.939631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 718.055824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.112595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.185837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 718.256657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.309057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.342974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 718.435543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.480566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.576738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 718.690223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.768381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.877353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 719.005478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.087289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.175531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.880792] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:14:07 (1787818447) [ 727.738726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg210-server [ 729.645905] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:14:11 (1787818451) [ 731.441293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg210-server [ 733.240934] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:14:15 (1787818455) [ 733.520413] LNet: 35870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.529305] LNet: 35870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 734.511620] Key type .llcrypt unregistered [ 734.516387] Key type ._llcrypt unregistered [ 741.742536] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 742.000414] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 742.011232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 742.020561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 744.859594] Key type ._llcrypt registered [ 744.862876] Key type .llcrypt registered [ 745.138615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.160522] alg: No test for adler32 (adler32-zlib) [ 746.082403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.144390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.173798] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 746.905891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 746.931935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 749.192955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 749.221494] LNet: Removed LNI 10.1.2.3@tcp [ 749.372401] LNet: 36595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.377750] LNetError: 36595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.390212] LNet: Removed LNI 192.168.202.10@tcp [ 750.142194] Key type .llcrypt unregistered [ 750.148264] Key type ._llcrypt unregistered [ 763.417590] Key type ._llcrypt registered [ 763.429177] Key type .llcrypt registered [ 765.031159] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:14:46 (1787818486) [ 765.564789] Key type .llcrypt unregistered [ 765.567853] Key type ._llcrypt unregistered [ 772.437881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 772.688023] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.696100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.702104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 775.452705] Key type ._llcrypt registered [ 775.455978] Key type .llcrypt registered [ 775.742769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.763835] alg: No test for adler32 (adler32-zlib) [ 776.772416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.835707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.852868] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 777.413830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 777.429790] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 779.795823] LNet: 38099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.801690] LNetError: 38099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 779.806267] LNet: Removed LNI 10.1.2.3@tcp [ 779.838326] LNet: Removed LNI 192.168.202.10@tcp [ 780.697489] Key type .llcrypt unregistered [ 780.699873] Key type ._llcrypt unregistered [ 795.082283] Key type ._llcrypt registered [ 795.084321] Key type .llcrypt registered [ 796.447493] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:15:18 (1787818518) [ 797.090317] Key type .llcrypt unregistered [ 797.092967] Key type ._llcrypt unregistered [ 805.218922] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 805.408932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 805.416326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.418848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.307190] Key type ._llcrypt registered [ 808.308627] Key type .llcrypt registered [ 808.556851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.586219] alg: No test for adler32 (adler32-zlib) [ 809.563776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.624963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 809.655504] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 811.998703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 812.022091] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.199090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 814.251240] LNet: Removed LNI 10.1.2.3@tcp [ 814.282147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 814.316472] LNet: Removed LNI 10.1.2.31@tcp [ 814.337471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 814.366484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.019851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 815.037737] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 816.037182] LNet: 40028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.040889] LNetError: 40028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 816.045278] LNet: Removed LNI 10.1.2.31@tcp [ 816.060748] LNet: Removed LNI 10.1.2.3@tcp [ 816.735138] Key type .llcrypt unregistered [ 816.745813] Key type ._llcrypt unregistered [ 830.923555] Key type ._llcrypt registered [ 830.927358] Key type .llcrypt registered [ 832.588740] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:15:54 (1787818554) [ 832.976194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.986439] alg: No test for adler32 (adler32-zlib) [ 833.901492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.970909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.009459] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 834.739840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.752895] LNet: 40986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.755920] LNetError: 40986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.774402] LNet: Removed LNI 192.168.202.10@tcp [ 834.785739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.836352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.10@tcp [ 834.850430] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 835.457570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.10@tcp [ 835.480956] LNet: Removed LNI 192.168.202.10@tcp [ 835.530974] LNet: 41121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.199478] Key type .llcrypt unregistered [ 836.201059] Key type ._llcrypt unregistered [ 850.329143] Key type ._llcrypt registered [ 850.339996] Key type .llcrypt registered [ 852.153444] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:16:13 (1787818573) [ 854.723208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg210-server [ 856.757755] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:16:18 (1787818578) [ 857.312378] Key type .llcrypt unregistered [ 857.317478] Key type ._llcrypt unregistered [ 864.826832] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 865.051359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 865.066295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 865.082937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 867.944788] Key type ._llcrypt registered [ 867.946153] Key type .llcrypt registered [ 868.320776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.386656] alg: No test for adler32 (adler32-zlib) [ 869.378992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.481941] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 870.401948] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 872.732744] LNet: 42967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.739651] LNetError: 42967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.744316] LNet: Removed LNI 192.168.202.10@tcp1 [ 872.758299] LNet: Removed LNI 10.1.2.3@tcp2 [ 873.512457] Key type .llcrypt unregistered [ 873.516987] Key type ._llcrypt unregistered [ 888.213780] Key type ._llcrypt registered [ 888.216705] Key type .llcrypt registered [ 889.991625] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:16:51 (1787818611) [ 890.438860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 890.469470] alg: No test for adler32 (adler32-zlib) [ 891.446789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.514861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.554722] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 892.490337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.521530] LNet: 43942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.533754] LNetError: 43942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.555055] LNet: Removed LNI 192.168.202.10@tcp [ 892.617524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.735722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 892.770843] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 900.390415] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:17:02 (1787818622) [ 900.579803] LNet: 44535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.597480] LNetError: 44535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.623923] LNet: Removed LNI 192.168.202.10@tcp [ 901.574420] Key type .llcrypt unregistered [ 901.577803] Key type ._llcrypt unregistered [ 909.299841] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.609239] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.623964] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.628928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.350288] Key type ._llcrypt registered [ 912.360927] Key type .llcrypt registered [ 912.613495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.627588] alg: No test for adler32 (adler32-zlib) [ 913.530802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.582961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.611974] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 914.338705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 914.351345] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 916.303644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.318468] LNet: 45282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.323280] LNetError: 45282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 916.327882] LNet: Removed LNI 10.1.2.3@tcp [ 916.340372] LNet: Removed LNI 192.168.202.10@tcp [ 916.359824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.449967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 916.479038] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 916.484301] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 917.180334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.209341] LNet: 45309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.217768] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 917.223835] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 917.228976] LNet: Removed LNI 10.1.2.3@tcp [ 917.242116] LNet: Removed LNI 192.168.202.10@tcp [ 917.262988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.339917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 917.358900] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 917.362473] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 917.756397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 917.777775] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 919.023448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.055986] LNet: 45471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.081748] LNetError: 45471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.098790] LNetError: 45471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 919.113896] LNet: Removed LNI 10.1.2.3@tcp [ 919.120221] LNet: Removed LNI 192.168.202.10@tcp [ 919.166827] LNet: Removed LNI 192.168.202.10@tcp2 [ 919.226370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.329385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 919.365707] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 919.370570] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.378943] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 926.796079] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:17:28 (1787818648) [ 926.899625] LNet: 46060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.912259] LNetError: 46060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.917406] LNetError: 46060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 926.927128] LNet: Removed LNI 10.1.2.3@tcp [ 926.931075] LNet: Removed LNI 192.168.202.10@tcp [ 926.943686] LNet: Removed LNI 192.168.202.10@tcp2 [ 927.592725] Key type .llcrypt unregistered [ 927.595156] Key type ._llcrypt unregistered [ 934.284550] Key type ._llcrypt registered [ 934.287394] Key type .llcrypt registered [ 934.558171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.577457] alg: No test for adler32 (adler32-zlib) [ 935.548927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.620236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.667831] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 937.023650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.046530] LNet: 46468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.054581] LNetError: 46468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.066571] LNet: Removed LNI 192.168.202.10@tcp [ 937.090850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.225871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 937.274983] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 938.053631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.079466] LNet: 46498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.082805] LNetError: 46498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.107746] LNet: Removed LNI 192.168.202.10@tcp [ 938.139956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.225429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 938.276354] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 938.939554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.978297] LNet: Removed LNI 192.168.202.10@tcp [ 939.015361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.108646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 939.143273] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 939.766075] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.862615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.865622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.869139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 941.234585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.274583] LNet: 46599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.277716] LNet: 46599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 941.289118] LNetError: 46599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.296727] LNetError: 46599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 941.318533] LNet: Removed LNI 192.168.202.10@tcp [ 941.344785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.411357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 941.436375] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 948.445566] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:17:50 (1787818670) [ 948.577182] LNet: 47188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.583854] LNetError: 47188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.609975] LNet: Removed LNI 192.168.202.10@tcp [ 949.478413] Key type .llcrypt unregistered [ 949.480685] Key type ._llcrypt unregistered [ 956.247060] Key type ._llcrypt registered [ 956.248451] Key type .llcrypt registered [ 956.607904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.627232] alg: No test for adler32 (adler32-zlib) [ 957.564898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.594107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.607505] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 958.496748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.510226] LNet: 47599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.513749] LNetError: 47599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.526927] LNet: Removed LNI 192.168.202.10@tcp [ 958.547744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.599825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 958.629284] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 965.286891] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:18:07 (1787818687) [ 965.522455] LNet: 48191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.529907] LNetError: 48191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.549211] LNet: Removed LNI 192.168.202.10@tcp [ 966.117484] Key type .llcrypt unregistered [ 966.118546] Key type ._llcrypt unregistered [ 971.466767] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 971.649505] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 971.651515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 971.654666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 974.086207] Key type ._llcrypt registered [ 974.089308] Key type .llcrypt registered [ 974.267555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.277105] alg: No test for adler32 (adler32-zlib) [ 975.146211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.201948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.229060] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 976.304411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.316610] LNet: 48646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.320898] LNetError: 48646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.330653] LNet: Removed LNI 192.168.202.10@tcp [ 976.358428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.422460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 976.443179] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 983.910888] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:18:25 (1787818705) [ 984.265807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.314357] LNet: 49245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.324528] LNetError: 49245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.359323] LNet: Removed LNI 192.168.202.10@tcp [ 984.406381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.547122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 991.889824] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:18:33 (1787818713) [ 992.026045] LNet: 49825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.587714] Key type .llcrypt unregistered [ 992.591384] Key type ._llcrypt unregistered [ 998.549446] Key type ._llcrypt registered [ 998.552026] Key type .llcrypt registered [ 998.687386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.710178] alg: No test for adler32 (adler32-zlib) [ 999.673033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.720895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 999.744892] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1000.826143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.869549] LNet: 50229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.887372] LNetError: 50229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.911336] LNet: Removed LNI 192.168.202.10@tcp [ 1000.940658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.041417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1001.098858] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1007.734746] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:18:50 (1787818730) [ 1007.974446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.010263] LNet: 50823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.021916] LNetError: 50823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.061570] LNet: Removed LNI 192.168.202.10@tcp [ 1008.088964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.167746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1014.300194] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:18:56 (1787818736) [ 1015.469712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1016.650714] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:18:59 (1787818739) [ 1018.110116] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1019.913995] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:19:01 (1787818741) [ 1021.521380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1023.115429] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:19:05 (1787818745) [ 1025.093443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1027.433760] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:19:08 (1787818748) [ 1028.855925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1030.743775] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:19:12 (1787818752) [ 1031.079700] LNet: 52875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.607580] Key type .llcrypt unregistered [ 1031.609415] Key type ._llcrypt unregistered [ 1038.174101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1038.357134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1038.372482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1038.377425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1041.113873] Key type ._llcrypt registered [ 1041.115824] Key type .llcrypt registered [ 1041.276845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.290628] alg: No test for adler32 (adler32-zlib) [ 1042.296777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.352633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.368916] LNet: Added LNI 192.168.202.10@tcp [8/512/0/180] [ 1042.372359] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1042.374921] LNet: 53198:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.530854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.610354] LNet: 53217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.622696] LNetError: 53217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.645532] LNet: Removed LNI 10.1.2.3@tcp [ 1042.652309] LNetError: 53217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.678692] LNet: Removed LNI 192.168.202.10@tcp [ 1042.711222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.843614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.893975] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1042.968828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.004259] LNet: 53242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.010594] LNetError: 53242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.029844] LNet: Removed LNI 192.168.202.10@tcp [ 1043.070386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.165964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.208707] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1043.211223] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.213119] LNet: 53250:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.278317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.296387] LNet: 53268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.300985] LNetError: 53268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.305681] LNet: Removed LNI 10.1.2.3@tcp [ 1043.307673] LNetError: 53268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.320566] LNet: Removed LNI 192.168.202.10@tcp [ 1043.344805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.436896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.493586] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1043.497562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.502596] LNet: 53276:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.728734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.755602] LNet: 53302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.764726] LNetError: 53302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.773630] LNet: Removed LNI 10.1.2.3@tcp [ 1043.775337] LNetError: 53302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.799350] LNet: Removed LNI 192.168.202.10@tcp [ 1043.871820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.941828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.979315] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 1043.987870] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1044.120788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.137489] LNet: 53330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.140630] LNetError: 53330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1044.143465] LNet: Removed LNI 10.1.2.3@tcp [ 1044.144396] LNetError: 53330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.167794] LNet: Removed LNI 192.168.202.10@tcp [ 1044.192250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.274328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1044.293271] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 1044.298211] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1044.302390] LNet: 53338:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1044.386966] LNet: Removed LNI 10.1.2.3@tcp [ 1044.396052] LNet: Removed LNI 192.168.202.10@tcp [ 1045.012887] Key type .llcrypt unregistered [ 1045.018356] Key type ._llcrypt unregistered [ 1059.973341] Key type ._llcrypt registered [ 1059.975388] Key type .llcrypt registered [ 1061.712875] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 04:19:43 (1787818783) [ 1063.697793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1066.085709] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:19:47 (1787818787) [ 1066.714488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.731507] alg: No test for adler32 (adler32-zlib) [ 1067.671324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.745493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1067.788837] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1068.478811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.10 [ 1068.493524] LNet: 54607:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1068.510159] LNet: Removed LNI 192.168.202.10@tcp [ 1075.342779] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:19:57 (1787818797) [ 1075.745472] LNet: 55180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.762552] LNetError: 55180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.788563] LNet: Removed LNI 192.168.202.10@tcp [ 1076.640673] Key type .llcrypt unregistered [ 1076.643309] Key type ._llcrypt unregistered [ 1084.048256] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1084.362121] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1086.108365] Key type ._llcrypt registered [ 1086.114863] Key type .llcrypt registered [ 1086.382348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.402792] alg: No test for adler32 (adler32-zlib) [ 1087.785760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1093.936688] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:20:16 (1787818816) [ 1094.008600] LNet: 56080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.016244] LNetError: 56080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1094.025837] LNet: Removed LNI 10.1.2.3@tcp [ 1094.562525] Key type .llcrypt unregistered [ 1094.563556] Key type ._llcrypt unregistered [ 1100.733113] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1101.019400] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1101.029781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1101.039984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1102.787580] Key type ._llcrypt registered [ 1102.790602] Key type .llcrypt registered [ 1102.979402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1102.998450] alg: No test for adler32 (adler32-zlib) [ 1104.051523] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1110.748805] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:20:32 (1787818832) [ 1112.584218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1114.341341] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:20:36 (1787818836) [ 1114.475654] LNet: 57262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.484438] LNetError: 57262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1114.506327] LNet: Removed LNI 10.1.2.3@tcp [ 1115.502661] Key type .llcrypt unregistered [ 1115.506682] Key type ._llcrypt unregistered [ 1121.960634] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1122.295600] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1122.299752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1122.308667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1125.353611] Key type ._llcrypt registered [ 1125.356544] Key type .llcrypt registered [ 1125.572666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.594513] alg: No test for adler32 (adler32-zlib) [ 1126.490870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.568551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.328337] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:20:57 (1787818857) [ 1135.761245] LNet: 58504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.766462] LNetError: 58504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1135.782337] LNet: Removed LNI 10.1.2.3@tcp [ 1136.748111] Key type .llcrypt unregistered [ 1136.749802] Key type ._llcrypt unregistered [ 1143.839147] Key type ._llcrypt registered [ 1143.843780] Key type .llcrypt registered [ 1144.100239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.118506] alg: No test for adler32 (adler32-zlib) [ 1145.051845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1145.103596] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1153.129229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1154.062942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1154.115260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1154.150674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1154.168453] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1154.339682] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1154.411691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.436274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.467996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.495291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.525705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.557938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.587205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.610912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.627771] LNet: 58945:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.655411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.690821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.731771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.776935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.844622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.888322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.943436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1155.580927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1155.639619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1155.692424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1155.720938] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1155.965706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.003284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.053383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.081288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.110513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.134229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.189808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.233786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.251198] LNet: 59055:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.271542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.300299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.343297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.413893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.449580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.490423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.532452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1157.227597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1157.266130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1157.305529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1157.320860] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1157.329792] LNet: Skipped 1 previous similar message [ 1157.614312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.644416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.682331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.725789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.760191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.806478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.872680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.925621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.947077] LNet: 59164:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.966625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.010683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.051281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.097051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.165466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.239548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.299195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1159.160936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1159.202708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1159.266646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1159.657630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.724409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.784800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.824431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.856031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.900677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.977815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1160.040505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1160.068653] LNet: 59273:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1160.103426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1160.154342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1160.204715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.287800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.316195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.343988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.378975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.004414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1161.043543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1161.089330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1161.110304] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1161.118747] LNet: Skipped 1 previous similar message [ 1161.447763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.504292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.553418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.589298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.638463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.682538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.745642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.788218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.840095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.863594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.900718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.954445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.010307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.039468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.065620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.619571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1162.661033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1162.709750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1162.981366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.010795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.047797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.084299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.115474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.157185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.193622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.228471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.259512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.306633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.331413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.355459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.388488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.421866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.457922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1163.809317] LNet: 59537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.818148] LNetError: 59537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.855930] LNet: Removed LNI 192.168.202.10@tcp [ 1164.247612] Key type .llcrypt unregistered [ 1164.249049] Key type ._llcrypt unregistered [ 1185.374403] Key type ._llcrypt registered [ 1185.379224] Key type .llcrypt registered [ 1186.808937] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:21:48 (1787818908) [ 1187.665323] Key type .llcrypt unregistered [ 1187.667332] Key type ._llcrypt unregistered [ 1194.656359] Key type ._llcrypt registered [ 1194.664114] Key type .llcrypt registered [ 1194.922897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1194.949953] alg: No test for adler32 (adler32-zlib) [ 1195.971593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1196.001770] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1205.430419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1218.398409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.465744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1218.499965] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1220.435435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1220.471781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1220.505130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1220.550163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1220.598100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1220.671256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1220.733280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1220.798213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1220.883085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1220.922847] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1220.935665] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1221.249235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.277431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1221.317317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.348108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.397044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.439792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.498867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.541189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.568656] LNet: 61027:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.592879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.626611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.721754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.787951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.853784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.945718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.985514] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1221.999443] LNet: Skipped 3 previous similar messages [ 1222.010600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1222.632822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1222.687450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1222.755820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1222.819722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1222.877782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1222.938765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1223.007910] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1223.013866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1223.073097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1223.126733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1223.521710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.566485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.615420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1223.648851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1223.686746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1223.731603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1223.774248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1223.819432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.857257] LNet: 61148:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.881476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.945751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.005186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.064197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.105437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.144247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.195280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1224.882330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1224.918975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1224.969086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1225.027147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1225.092409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1225.150790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1225.209360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1225.317167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1225.387869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1225.429799] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1225.434067] LNet: Skipped 3 previous similar messages [ 1225.726123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.758218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.803904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.864093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.904481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.945892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.982852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.018647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.029744] LNet: 61270:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.052780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.086295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.118938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.149828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.220945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.270960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.336284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1226.946783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1227.009597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1227.036878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1227.064808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1227.096910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1227.152931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1227.253408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1227.356589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1227.423327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1227.822827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.860622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.898408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.943955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.990767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.027415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.087064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.134674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.174949] LNet: 61391:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.243705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.295738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.361837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.436419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.520847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.583949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.642668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.243350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1229.313093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1229.365403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1229.419422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1229.462967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1229.515099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1229.568995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1229.627446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1229.686774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1229.712774] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1229.715914] LNet: Skipped 6 previous similar messages [ 1230.023473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.052644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.079447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.111151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.145515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.192435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.230497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.254152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.309952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.377981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.416895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.473577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.505435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.580937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.655629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1231.306882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1231.345702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1231.391335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1231.425321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1231.465277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1231.516523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1231.573947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1231.654253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1231.709791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1232.094969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.150912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.199857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.269844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.331218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.410162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.514271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.621339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.663040] LNet: 61634:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.675705] LNet: 61634:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1232.698645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.741416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.796910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.858950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.898954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.949597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.012342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.639641] LNet: 61714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.644868] LNetError: 61714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.652668] LNet: Removed LNI 192.168.202.10@tcp [ 1239.726396] LNet: Removed LNI 192.168.202.10@tcp1 [ 1240.256700] Key type .llcrypt unregistered [ 1240.263634] Key type ._llcrypt unregistered [ 1261.621275] Key type ._llcrypt registered [ 1261.624188] Key type .llcrypt registered [ 1263.183718] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:23:04 (1787818984) [ 1264.452553] Key type .llcrypt unregistered [ 1264.455881] Key type ._llcrypt unregistered [ 1272.083967] Key type ._llcrypt registered [ 1272.086026] Key type .llcrypt registered [ 1272.432821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.474428] alg: No test for adler32 (adler32-zlib) [ 1273.444477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1273.468547] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1282.393953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1283.077360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1283.103846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1283.125139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1283.140531] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1283.362487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.414060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.457854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.493422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.519685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.558771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.595639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.635189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.668361] LNet: 62970:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.707345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.771539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.823841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.854553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.889934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.951885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.987868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1284.678392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1284.720675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1284.756972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1284.774911] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1285.038587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1285.063536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1285.092541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.116332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.157803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.192468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1285.221704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1285.258705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.278887] LNet: 63079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1285.300369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.346654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.382766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.414872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.444802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.488473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.520402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1286.092407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1286.122945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1286.151479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1286.169922] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1286.407376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.439585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.469922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.512718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1286.564700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.613651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.662923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.715783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.746352] LNet: 63188:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.791485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.836815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.913234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.978789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.058510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.091962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.124277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1287.504578] LNet: 63233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.512047] LNetError: 63233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.528882] LNet: Removed LNI 192.168.202.10@tcp [ 1288.101895] Key type .llcrypt unregistered [ 1288.103546] Key type ._llcrypt unregistered [ 1306.991493] Key type ._llcrypt registered [ 1306.995104] Key type .llcrypt registered [ 1308.395220] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:23:50 (1787819030) [ 1309.100277] Key type .llcrypt unregistered [ 1309.102213] Key type ._llcrypt unregistered [ 1315.842553] Key type ._llcrypt registered [ 1315.845320] Key type .llcrypt registered [ 1316.035348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1316.065214] alg: No test for adler32 (adler32-zlib) [ 1317.097162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1317.121235] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1325.172647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1336.227621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.295041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.337873] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1337.987384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1338.032989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1338.072562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1338.113780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1338.161822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1338.208864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1338.289992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1338.368749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1338.440926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1338.463638] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1338.469633] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1338.704823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.753457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.783471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.818397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.848371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.885955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.910985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.936530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.951273] LNet: 64706:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.966578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.994608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.019232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.052468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.086967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.115937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.148896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1339.186969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.747602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1339.799960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1339.846985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1339.903649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1339.965178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1340.009897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1340.073606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1340.138838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1340.198604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1340.257801] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1340.263104] LNet: Skipped 2 previous similar messages [ 1340.535928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.568206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1340.604774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.638890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.667809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.697459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1340.734113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1340.772082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1340.793956] LNet: 64830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1340.824481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1340.867855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1340.924060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1340.981552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.026364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.068296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.121969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.178257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.790226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1341.860739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1341.947944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1341.994541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1342.059920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1342.113963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1342.155585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1342.192795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1342.242516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1342.273493] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1342.557742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.585595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.611265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.673422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.726617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.767428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.834831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.882821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.909686] LNet: 64953:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.946056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.988775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.014487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.081983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.144169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.194414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.239899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.271098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.636833] LNet: 65035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.647396] LNetError: 65035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.667490] LNet: Removed LNI 192.168.202.10@tcp [ 1348.678926] LNet: Removed LNI 192.168.202.10@tcp1 [ 1349.053321] Key type .llcrypt unregistered [ 1349.054395] Key type ._llcrypt unregistered [ 1368.181422] Key type ._llcrypt registered [ 1368.183577] Key type .llcrypt registered [ 1369.483837] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:24:51 (1787819091) [ 1371.484560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1373.431585] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:24:55 (1787819095) [ 1374.132558] Key type .llcrypt unregistered [ 1374.135287] Key type ._llcrypt unregistered [ 1382.269910] Key type ._llcrypt registered [ 1382.272912] Key type .llcrypt registered [ 1382.695749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1382.735751] alg: No test for adler32 (adler32-zlib) [ 1383.766642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1383.801413] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1393.300407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1393.805629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1393.852481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1393.892448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1393.926607] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1394.081296] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1394.185739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1394.225907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1394.256435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1394.288967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1394.327781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1394.369574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1394.415521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1394.493584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1394.544330] LNet: 66608:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1394.578243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1394.638402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1394.687487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1394.737264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1394.781288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1394.835601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1395.135490] LNet: 66651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.140866] LNetError: 66651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.160369] LNet: Removed LNI 192.168.202.10@tcp [ 1395.572839] Key type .llcrypt unregistered [ 1395.575862] Key type ._llcrypt unregistered [ 1415.787463] Key type ._llcrypt registered [ 1415.790051] Key type .llcrypt registered [ 1415.984795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1416.003431] alg: No test for adler32 (adler32-zlib) [ 1416.967361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1416.992337] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1425.414866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1436.802475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.886518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.931905] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1438.467560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1438.516903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1438.588102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1438.649889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1438.702588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1438.793540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1438.889556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1438.951748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1439.023964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1439.066969] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1439.417820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1439.479221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1439.518676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1439.526629] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1439.536104] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1439.605082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1439.655937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1439.694203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1439.744070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1439.871181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1439.911886] LNet: 67538:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1439.947750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.024589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.072103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.114592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.165320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.220587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1446.499985] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1446.960305] LNet: 67616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.966995] LNetError: 67616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.031739] LNet: Removed LNI 192.168.202.10@tcp [ 1448.043489] LNetError: 67616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.061557] LNet: Removed LNI 192.168.202.10@tcp1 [ 1448.634714] Key type .llcrypt unregistered [ 1448.639789] Key type ._llcrypt unregistered [ 1470.003615] Key type ._llcrypt registered [ 1470.008693] Key type .llcrypt registered [ 1471.559352] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:26:33 (1787819193) [ 1471.850457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.865227] alg: No test for adler32 (adler32-zlib) [ 1472.800714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.878580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.916078] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1474.042581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.071617] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1475.259636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1475.393385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1475.496709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1475.570211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1475.625030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1475.704530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1475.774849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1475.802573] LNetError: 68719:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1475.935464] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1475.936186] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1477.983624] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1478.419501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1481.055605] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1482.124698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1482.165771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1482.231390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.261360] LNet: 68816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.266711] LNetError: 68816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.295390] LNet: Removed LNI 192.168.202.10@tcp [ 1483.298714] LNetError: 68816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.332545] LNet: Removed LNI 192.168.202.10@tcp1 [ 1483.373578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.454538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.471472] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1484.526852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1484.548038] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1485.610691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1485.640350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.712777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1485.775269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1485.820755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1485.894799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1485.953803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1486.001925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1486.025596] LNetError: 69089:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1486.495358] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1489.745486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1492.646148] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1492.660437] LNet: Skipped 1 previous similar message [ 1497.030790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1497.113807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1497.186636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1505.450561] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:27:07 (1787819227) [ 1505.546946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.588782] LNet: 69790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.600287] LNetError: 69790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.616610] LNet: Removed LNI 192.168.202.10@tcp [ 1505.654125] LNet: Removed LNI 192.168.202.10@tcp1 [ 1505.718310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.779220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.800986] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1506.619754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1506.645366] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1507.595938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1507.687974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1507.731933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1507.777682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1507.838704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1507.875985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1507.918752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1508.013436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1508.831323] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.10@tcp [ 1508.831871] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1508.849347] LNet: Skipped 1 previous similar message [ 1522.673498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1522.899811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1522.936319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1522.962336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 500 [ 1523.098890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1528.842647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1528.945055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.977273] LNet: 70274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.987774] LNetError: 70274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.994465] LNetError: 70274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1529.010924] LNet: Removed LNI 192.168.202.10@tcp [ 1529.025881] LNet: Removed LNI 192.168.202.10@tcp1 [ 1529.099184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.208794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1529.232676] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1529.867969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1529.882593] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1530.544344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1530.580260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1530.642417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1530.706842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1530.752070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1530.801760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1530.881261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1530.919684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1531.231418] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1531.239149] LNet: Skipped 3 previous similar messages [ 1532.299265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.812888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.853896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1541.895184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.148954] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:27:51 (1787819271) [ 1549.751374] LNet: 71310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.757191] LNetError: 71310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.760961] LNetError: 71310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1549.774532] LNet: Removed LNI 192.168.202.10@tcp [ 1549.790272] LNet: Removed LNI 192.168.202.10@tcp1 [ 1550.515102] Key type .llcrypt unregistered [ 1550.518030] Key type ._llcrypt unregistered [ 1557.325914] Key type ._llcrypt registered [ 1557.329178] Key type .llcrypt registered [ 1557.597958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1557.621542] alg: No test for adler32 (adler32-zlib) [ 1558.620524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1558.666669] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1567.588487] Lustre: DEBUG MARKER: Initial discovery [ 1567.780989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1570.901050] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1571.188940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1571.229885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1571.248428] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1571.268355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1571.298616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -l 3 [ 1571.335675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -e local_error [ 1571.372598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m PUT -l 6 [ 1572.013508] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1572.777571] Lustre: DEBUG MARKER: Force oleg210-client.virtnet to discover 192.168.202.110@tcp (in background) [ 1572.916051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1574.535396] Lustre: DEBUG MARKER: Force oleg210-server to discover 192.168.202.10@tcp [ 1576.032791] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1577.233511] Lustre: DEBUG MARKER: Wait for 72074 [ 1578.844684] Lustre: DEBUG MARKER: Finished wait on 72074 [ 1578.993666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1579.057861] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1579.072677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1579.440604] LNet: 72532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.451035] LNetError: 72532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.468813] LNet: Removed LNI 192.168.202.10@tcp [ 1579.883897] Key type .llcrypt unregistered [ 1579.886300] Key type ._llcrypt unregistered [ 1599.325395] Key type ._llcrypt registered [ 1599.327612] Key type .llcrypt registered [ 1600.861454] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:28:42 (1787819322) [ 1601.536312] Key type .llcrypt unregistered [ 1601.538757] Key type ._llcrypt unregistered [ 1609.455872] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1609.611109] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1609.620926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1609.624575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1612.391000] Key type ._llcrypt registered [ 1612.395550] Key type .llcrypt registered [ 1612.608417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1612.642595] alg: No test for adler32 (adler32-zlib) [ 1613.635157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.718722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.745670] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1614.688338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1614.711724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1625.130326] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:29:07 (1787819347) [ 1625.453974] LNet: 74614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.460965] LNetError: 74614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1625.473033] LNet: Removed LNI 10.1.2.3@tcp [ 1625.493356] LNet: Removed LNI 192.168.202.10@tcp [ 1626.417443] Key type .llcrypt unregistered [ 1626.424369] Key type ._llcrypt unregistered [ 1635.145668] Key type ._llcrypt registered [ 1635.148826] Key type .llcrypt registered [ 1635.316454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1635.335057] alg: No test for adler32 (adler32-zlib) [ 1636.362814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.453394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.486609] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1637.514368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1637.541930] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1638.832318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.10@tcp [ 1639.904289] LNetError: Unexpected error -22 connecting to 192.168.202.10@tcp at host 192.168.202.10:988 [ 1646.179808] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:29:27 (1787819367) [ 1646.294773] LNet: 75743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.303853] LNetError: 75743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1646.317744] LNet: Removed LNI 10.1.2.3@tcp [ 1646.349330] LNet: Removed LNI 192.168.202.10@tcp [ 1647.209438] Key type .llcrypt unregistered [ 1647.219939] Key type ._llcrypt unregistered [ 1654.947891] Key type ._llcrypt registered [ 1654.951613] Key type .llcrypt registered [ 1655.231856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1655.255802] alg: No test for adler32 (adler32-zlib) [ 1656.183459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.245484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.282208] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1657.129803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1657.152906] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 1658.282817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.10@tcp1 --nid 192.168.202.10@tcp2 [ 1663.398640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp2 [ 1663.418700] LNetError: 76589:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.10@tcp1 to 192.168.202.10@tcp2 - Route table may be misconfigured [ 1663.437569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp2 192.168.202.10@tcp1 [ 1671.188537] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:29:52 (1787819392) [ 1671.308794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.366319] LNet: 77155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.377163] LNetError: 77155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1671.385512] LNet: Removed LNI 192.168.202.10@tcp1 [ 1672.427081] LNet: Removed LNI 192.168.202.10@tcp2 [ 1672.486722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.574495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1672.613201] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1673.469753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1673.496488] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1674.506919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1674.549495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.10@tcp [ 1674.590597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp1 [ 1674.665990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp -e network_timeout [ 1674.730568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -e network_timeout [ 1674.788692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -e network_timeout [ 1674.862969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -e network_timeout [ 1674.921169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1674.959451] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1674.988641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1675.063695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1675.204976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1675.621829] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1682.738405] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:30:04 (1787819404) [ 1682.911576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.944887] LNet: 78021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.951467] LNetError: 78021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.959207] LNetError: 78021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1682.974273] LNet: Removed LNI 192.168.202.10@tcp [ 1684.006633] LNet: Removed LNI 192.168.202.10@tcp1 [ 1684.074388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.186710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1684.458461] LNet: 78065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.767172] Key type .llcrypt unregistered [ 1684.771630] Key type ._llcrypt unregistered [ 1698.380842] Key type ._llcrypt registered [ 1698.386956] Key type .llcrypt registered [ 1700.100890] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:30:21 (1787819421) [ 1700.619791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.636919] alg: No test for adler32 (adler32-zlib) [ 1701.624653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.436856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1705.106327] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:30:27 (1787819427) [ 1705.241584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.277539] LNet: 79157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.320409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.436390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.512904] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1706.239944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1706.263110] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1707.512641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1707.597665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp1 [ 1707.645615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp1 [ 1716.034284] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:30:37 (1787819437) [ 1718.201203] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg210-server [ 1720.292631] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:30:41 (1787819441) [ 1722.779614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg210-server [ 1725.076465] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:30:46 (1787819446) [ 1727.881466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg210-server [ 1730.392580] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:30:51 (1787819451) [ 1732.820703] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg210-server [ 1735.325072] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:30:56 (1787819456) [ 1737.825272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg210-server [ 1739.580688] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:31:01 (1787819461) [ 1741.794949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg210-server [ 1744.167554] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:31:05 (1787819465) [ 1746.071797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg210-server [ 1747.771396] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:31:09 (1787819469) [ 1750.015199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg210-server [ 1752.338911] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:31:14 (1787819474) [ 1754.715900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg210-server [ 1757.044121] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:31:18 (1787819478) [ 1757.259515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.324388] LNet: 83039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.334076] LNetError: 83039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.370747] LNet: Removed LNI 192.168.202.10@tcp [ 1757.423110] LNet: Removed LNI 192.168.202.10@tcp1 [ 1757.500218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.543841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.561648] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1758.368603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1758.533430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1758.737036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.765160] LNet: 83191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.782267] LNetError: 83191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.785223] LNetError: 83191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1758.819795] LNet: Removed LNI 192.168.202.10@tcp [ 1758.858423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.955600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1758.984645] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1760.208389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1760.496388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.515408] LNetError: 83336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.577890] LNet: Removed LNI 192.168.202.10@tcp [ 1761.624757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.739897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.787982] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1762.730254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1762.829171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1763.135386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.176181] LNet: 83484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.182413] LNet: 83484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1763.196840] LNetError: 83484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.282911] LNet: Removed LNI 192.168.202.10@tcp [ 1764.335437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.431869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1764.475461] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1765.367954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1766.700754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.783758] LNet: Removed LNI 192.168.202.10@tcp [ 1767.838833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.905385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.953161] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1768.975924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1769.054150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1769.296669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.331861] LNet: 83785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.337743] LNet: 83785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1769.344246] LNetError: 83785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.349735] LNetError: 83785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1769.367602] LNet: Removed LNI 192.168.202.10@tcp [ 1769.408324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.480185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1769.511425] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1770.407292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1770.684969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.755915] LNet: Removed LNI 192.168.202.10@tcp [ 1771.810614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.893545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1771.915546] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1772.665515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1772.751880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1774.226692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.345205] LNet: Removed LNI 192.168.202.10@tcp [ 1775.431866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.547395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1775.589398] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1776.657988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1778.150242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.174812] LNet: 84242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.180704] LNet: 84242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1778.188889] LNetError: 84242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.197552] LNetError: 84242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1778.222274] LNet: Removed LNI 192.168.202.10@tcp [ 1778.244883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.338993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.388136] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1779.518525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1779.631200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1781.141863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.183903] LNet: Removed LNI 192.168.202.10@tcp [ 1782.249748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.292402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1782.334770] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1783.491917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1785.239111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.344390] LNet: Removed LNI 192.168.202.10@tcp [ 1786.459793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.595459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.641622] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1787.871785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1787.982457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1789.508604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.635885] LNet: Removed LNI 192.168.202.10@tcp [ 1790.715115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.812946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1790.853610] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1791.958479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1793.440876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.533126] LNet: Removed LNI 192.168.202.10@tcp [ 1794.571785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.718859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.786914] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1795.825212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1795.916748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.973356] LNet: 84996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.982541] LNet: 84996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1796.010757] LNetError: 84996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.027758] LNetError: 84996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1796.043064] LNet: Removed LNI 192.168.202.10@tcp [ 1796.101938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.192428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.255900] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1797.315635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1797.421564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.464878] LNet: Removed LNI 192.168.202.10@tcp [ 1797.490766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.568339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1797.704073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.782365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.887747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1797.995665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.047962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.108754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1798.243831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.296504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.375365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1798.411776] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1799.368496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.404360] LNet: Removed LNI 192.168.202.10@tcp [ 1799.430922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.512456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1806.267547] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:32:08 (1787819528) [ 1806.459261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.513847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.593268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1806.612342] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1807.679407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.739459] LNet: Removed LNI 192.168.202.10@tcp231 [ 1807.777598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.902442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1807.936985] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1807.950653] LNet: 86031:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.429471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1808.468957] LNet: Removed LNI 192.168.202.10@tcp231 [ 1808.497744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1808.525317] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1809.830050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.910050] LNet: Removed LNI 192.168.202.10@tcp231 [ 1809.942304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.018341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1810.071255] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 1810.082919] LNet: 86184:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1810.594166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1810.662576] LNet: Removed LNI 192.168.202.10@tcp231 [ 1810.689273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1810.721989] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 1812.553983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.634541] LNet: Removed LNI 192.168.202.10@tcp231 [ 1812.672740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.836219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1812.890293] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 1812.903377] LNet: 86335:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.584496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1813.610337] LNet: Removed LNI 192.168.202.10@tcp231 [ 1813.686641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1813.773808] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 1815.739831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.800446] LNet: Removed LNI 192.168.202.10@tcp231 [ 1815.868521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.004975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1816.064623] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1816.077724] LNet: 86487:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1816.825900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1816.872229] LNet: Removed LNI 192.168.202.10@tcp231 [ 1816.885221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1816.928188] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1826.493576] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:32:28 (1787819548) [ 1826.717428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.809150] LNet: Removed LNI 192.168.202.10@tcp231 [ 1826.848190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.936086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1826.955429] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1827.864252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1827.940696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1827.969804] LNet: Removed LNI 192.168.202.10@tcp232 [ 1827.987992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1828.013751] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1829.029236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1829.075838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1829.119577] LNet: Removed LNI 192.168.202.10@tcp232 [ 1829.159849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1829.200965] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1830.000789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1830.050631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1830.089455] LNet: Removed LNI 192.168.202.10@tcp232 [ 1830.110258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1830.136993] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1830.923714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1830.962564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1831.002581] LNet: Removed LNI 192.168.202.10@tcp232 [ 1831.041549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1837.802766] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:32:39 (1787819559) [ 1838.268883] LNet: 88325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.272919] LNet: 88325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1839.152152] Key type .llcrypt unregistered [ 1839.153617] Key type ._llcrypt unregistered [ 1846.689443] Key type ._llcrypt registered [ 1846.691196] Key type .llcrypt registered [ 1847.057053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1847.093180] alg: No test for adler32 (adler32-zlib) [ 1848.084386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1848.117310] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1856.694515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1866.604193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.662304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1866.684472] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1868.217162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 1868.239971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 1868.298358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 1868.339191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 1868.373535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 1868.419193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 1868.469354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 1868.509544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 1868.566527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1868.605098] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1868.617330] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1869.005276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1869.072247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1869.119285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1869.165327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1869.207032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1869.256420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1869.300918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1869.333448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1869.356257] LNet: 88992:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1869.376143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1869.445709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1869.497240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1869.539774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1869.571337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1869.604412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1870.345213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 1870.410671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 1870.465694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 1870.545454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 1870.608454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 1870.654991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 1870.727396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 1870.800318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 1870.849333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1870.886959] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1871.396827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1871.468696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1871.520455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1871.572833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1871.685968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1871.743902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1871.814718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1871.867390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1871.897351] LNet: 89112:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1871.913898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1871.961855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1872.012462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1872.050357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1872.079251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1872.111712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1872.674398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 1872.719416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 1872.780327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 1872.823115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 1872.875390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 1872.906937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 1872.941197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 1872.974349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 1873.008082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1873.031392] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1873.270040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1873.302554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1873.331956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1873.357684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1873.383957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1873.408527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1873.447489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1873.484721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1873.503866] LNet: 89232:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1873.528780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1873.572389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1873.602978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1873.640595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1873.676326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1873.725959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1874.435742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 1874.509251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 1874.574144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 1874.652298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 1874.729818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 1874.807895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 1874.862335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 1874.905992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 1874.944376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1875.306191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1875.334994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1875.376899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1875.407630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1875.441326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1875.476982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1875.515484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1875.559589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1875.575832] LNet: 89352:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1875.594946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1875.645591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1875.675079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1875.709925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1875.755427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1875.784277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1876.433819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 1876.469054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 1876.503851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 1876.556888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 1876.630710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 1876.736749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 1876.790113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 1876.848324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 1876.922927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1876.963765] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1876.968280] LNet: Skipped 1 previous similar message [ 1877.306729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1877.350369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1877.415791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1877.472603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1877.511845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1877.554327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1877.609582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1877.682423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1877.727489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1877.772215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1877.825279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1877.889816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1877.940725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1878.002782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1878.567521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 1878.591895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 1878.635750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 1878.714880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 1878.784959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 1878.851211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 1878.898207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 1878.936751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 1878.980907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1879.012967] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1879.312181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1879.351880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1879.377866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1879.406997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1879.431171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1879.453522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1879.481957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1879.526078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1879.578758] LNet: 89592:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1879.590118] LNet: 89592:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1879.634575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1879.735724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1879.791591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1879.836126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1879.876141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1879.938344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1886.580239] LNet: 89671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.588395] LNetError: 89671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.595673] LNet: Removed LNI 192.168.202.10@tcp [ 1886.610481] LNet: Removed LNI 192.168.202.10@tcp1 [ 1887.071679] Key type .llcrypt unregistered [ 1887.075066] Key type ._llcrypt unregistered [ 1906.975449] Key type ._llcrypt registered [ 1906.981971] Key type .llcrypt registered [ 1908.347260] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:33:50 (1787819630) [ 1908.918093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.942986] alg: No test for adler32 (adler32-zlib) [ 1909.873739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.941528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.967654] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1910.731893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1910.762726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1910.781436] LNetError: 90651:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1910.848844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.10@tcp [ 1910.946996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1918.049814] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:34:00 (1787819640) [ 1918.303975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.322965] LNet: 91241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.347470] LNetError: 91241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.371761] LNet: Removed LNI 192.168.202.10@tcp [ 1918.405747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.495464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.547686] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1919.351839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1919.368508] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1920.450646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.10@tcp [ 1920.469869] LNetError: 91508:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1920.480585] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1920.491624] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1920.640594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.10@tcp [ 1929.533874] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:34:11 (1787819651) [ 1932.316604] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg210-server [ 1934.865638] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:34:16 (1787819656) [ 1937.051402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg210-server [ 1938.383293] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:34:20 (1787819660) [ 1940.220388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg210-server [ 1942.766389] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:34:24 (1787819664) [ 1943.010244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.067216] LNet: 93130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.087626] LNetError: 93130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.099654] LNet: Removed LNI 192.168.202.10@tcp [ 1943.138115] LNet: Removed LNI 192.168.202.10@tcp1 [ 1943.171352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.328339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.378917] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1944.406522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1944.486291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1944.558630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1944.732441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1944.832562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1944.921467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1952.265693] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:34:33 (1787819673) [ 1952.385576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.412790] LNet: 93856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.418899] LNetError: 93856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.429977] LNetError: 93856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1952.457195] LNet: Removed LNI 192.168.202.10@tcp [ 1952.484359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.577412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.597938] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1960.315641] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:34:42 (1787819682) [ 1960.449147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.492342] LNet: 94560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.499670] LNetError: 94560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.535905] LNet: Removed LNI 192.168.202.10@tcp [ 1960.598643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.740862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.790425] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1968.552842] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:34:50 (1787819690) [ 1970.457224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1972.616274] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:34:54 (1787819694) [ 1973.326635] LNet: 95601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.331795] LNetError: 95601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.345192] LNet: Removed LNI 192.168.202.10@tcp [ 1974.095176] Key type .llcrypt unregistered [ 1974.099680] Key type ._llcrypt unregistered [ 1981.418221] Key type ._llcrypt registered [ 1981.419718] Key type .llcrypt registered [ 1981.741693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.779880] alg: No test for adler32 (adler32-zlib) [ 1982.820452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1982.881524] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1991.623568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1997.122631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.110@tcp [ 1997.499302] LNet: 96027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.503945] LNetError: 96027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.520383] LNet: Removed LNI 192.168.202.10@tcp [ 1997.928582] Key type .llcrypt unregistered [ 1997.930215] Key type ._llcrypt unregistered [ 2009.030189] Key type ._llcrypt registered [ 2009.033224] Key type .llcrypt registered [ 2010.355790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2011.936918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2013.688957] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:35:35 (1787819735) [ 2015.124504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2016.674938] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:35:38 (1787819738) [ 2018.227965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg210-server [ 2019.659966] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:35:41 (1787819741) [ 2021.560926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg210-server [ 2023.310657] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:35:45 (1787819745) [ 2025.300240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2025.355475] alg: No test for adler32 (adler32-zlib) [ 2026.602117] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2026.824564] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2028.479234] Key type lgssc registered [ 2029.864343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.290971] Key type lgssc unregistered [ 2050.649603] LNet: 99205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.660684] LNetError: 99205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.697652] LNet: Removed LNI 192.168.202.10@tcp [ 2051.618207] Key type .llcrypt unregistered [ 2051.621251] Key type ._llcrypt unregistered [ 2066.468364] Key type ._llcrypt registered [ 2066.470438] Key type .llcrypt registered [ 2068.208277] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:36:29 (1787819789) [ 2068.800179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.817901] alg: No test for adler32 (adler32-zlib) [ 2069.826196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.943324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.017075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.050993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2070.087634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2078.893185] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:36:40 (1787819800) [ 2079.098794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.140592] LNet: 100619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.183684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.271674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.330454] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2080.729853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.779111] LNet: 100764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.806229] LNetError: 100764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.827343] LNet: Removed LNI 192.168.202.10@tcp [ 2080.858356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.933842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2080.958879] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2081.331551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.352613] LNetError: 100792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.370573] LNet: Removed LNI 192.168.202.10@tcp [ 2081.387815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.443427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2081.468838] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2090.669163] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:36:52 (1787819812) [ 2090.838850] LNet: 101376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.854323] LNet: 101376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2090.866961] LNetError: 101376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.906822] LNet: Removed LNI 192.168.202.10@tcp [ 2092.306680] Key type .llcrypt unregistered [ 2092.314362] Key type ._llcrypt unregistered [ 2101.787688] Key type ._llcrypt registered [ 2101.791933] Key type .llcrypt registered [ 2102.102812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.136737] alg: No test for adler32 (adler32-zlib) [ 2103.263287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.382498] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2103.385258] LNet: *** cfs_fail_loc=e003, val=100*** [ 2104.300466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2104.346836] LNet: *** cfs_fail_loc=e003, val=100*** [ 2104.356270] LNet: Skipped 2 previous similar messages [ 2104.366205] Ping reply received with mapping: [ 2104.367758] found metadata reply NI[192.168.202.10@tcp] [ 2104.370527] 0a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2104.373715] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2104.376889] 5a 5a 5a 5a 5a 5a 5a 5a [ 2104.623443] LNet: 101785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.628338] LNetError: 101785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.730047] LNet: Removed LNI 192.168.202.10@tcp [ 2106.540529] Key type .llcrypt unregistered [ 2106.543543] Key type ._llcrypt unregistered [ 2120.571838] Key type ._llcrypt registered [ 2120.574264] Key type .llcrypt registered [ 2122.010739] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:37:23 (1787819843) [ 2122.533307] Key type .llcrypt unregistered [ 2122.535241] Key type ._llcrypt unregistered [ 2130.484974] Key type ._llcrypt registered [ 2130.489537] Key type .llcrypt registered [ 2130.696633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2130.709417] alg: No test for adler32 (adler32-zlib) [ 2131.870251] LNetError: 102843:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2132.450459] Key type .llcrypt unregistered [ 2132.454057] Key type ._llcrypt unregistered [ 2138.148667] Key type ._llcrypt registered [ 2138.151688] Key type .llcrypt registered [ 2139.476779] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:37:41 (1787819861) [ 2140.161392] Key type .llcrypt unregistered [ 2140.163886] Key type ._llcrypt unregistered [ 2148.101210] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2148.108912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2150.892297] Key type ._llcrypt registered [ 2150.895844] Key type .llcrypt registered [ 2151.075594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2151.088350] alg: No test for adler32 (adler32-zlib) [ 2152.039733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.138720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2162.580124] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:38:04 (1787819884) [ 2162.839664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.880901] LNet: 104631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.887351] LNetError: 104631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2162.905534] LNet: Removed LNI 10.1.2.3@tcp [ 2162.944141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.968466] Key type .llcrypt unregistered [ 2163.972582] Key type ._llcrypt unregistered [ 2170.456200] Key type ._llcrypt registered [ 2170.457897] Key type .llcrypt registered [ 2170.776974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2170.819417] alg: No test for adler32 (adler32-zlib) [ 2171.773973] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2171.842817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2172.007773] LNet: 104924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.032312] LNetError: 104924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.053397] LNet: Removed LNI 192.168.202.10@tcp [ 2172.651634] Key type .llcrypt unregistered [ 2172.657540] Key type ._llcrypt unregistered [ 2185.245555] Key type ._llcrypt registered [ 2185.247931] Key type .llcrypt registered [ 2186.503392] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:38:28 (1787819908) [ 2187.322577] Key type .llcrypt unregistered [ 2187.325835] Key type ._llcrypt unregistered [ 2194.921364] Key type ._llcrypt registered [ 2194.927744] Key type .llcrypt registered [ 2195.197425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2195.215719] alg: No test for adler32 (adler32-zlib) [ 2203.580454] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:38:45 (1787819925) [ 2203.727132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.789245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.877987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2203.923379] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2204.840776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.880435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2204.909493] LNet: Removed LNI 192.168.202.10@tcp [ 2205.209205] LNet: 106681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.670268] Key type .llcrypt unregistered [ 2205.674858] Key type ._llcrypt unregistered [ 2216.121364] Key type ._llcrypt registered [ 2216.122532] Key type .llcrypt registered [ 2217.305061] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:38:59 (1787819939) [ 2217.768995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.804235] alg: No test for adler32 (adler32-zlib) [ 2218.788834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.901642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2218.950090] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2220.043946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2220.077145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.10@tcp [ 2220.096725] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2220.102164] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2220.109352] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2220, 0, 0 [ 2220.116543] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2220.122786] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2220.125119] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2220.131036] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2220, 0, 0 [ 2220.140829] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2220.143274] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2220.147134] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2220.148655] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2220, 0, 2270 [ 2220.154899] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2220.157324] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2220.166291] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2220.171297] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2220, 0, 2270 [ 2220.176766] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2228.344379] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:39:10 (1787819950) [ 2229.138052] LNet: 108222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.151814] LNetError: 108222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.172691] LNet: Removed LNI 192.168.202.10@tcp [ 2230.188476] Key type .llcrypt unregistered [ 2230.192195] Key type ._llcrypt unregistered [ 2239.822542] Key type ._llcrypt registered [ 2239.823899] Key type .llcrypt registered [ 2240.264496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2240.316314] alg: No test for adler32 (adler32-zlib) [ 2241.306936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2241.346218] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2250.488798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2261.661413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.708859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2261.739233] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2262.765788] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2262.910269] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2262.918056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2262.920609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2265.421691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2265.441695] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2275.154091] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:39:57 (1787819997) [ 2275.557561] LNet: 109736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.580626] LNetError: 109736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.589987] LNet: Removed LNI 192.168.202.10@tcp [ 2276.639397] LNet: Removed LNI 192.168.202.10@tcp1 [ 2276.644686] LNetError: 109736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2276.655978] LNetError: 109736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2276.688677] LNet: Removed LNI 10.1.2.3@tcp99 [ 2277.546763] Key type .llcrypt unregistered [ 2277.549638] Key type ._llcrypt unregistered [ 2284.722592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2284.731901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2287.372667] Key type ._llcrypt registered [ 2287.380383] Key type .llcrypt registered [ 2287.679641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2287.705635] alg: No test for adler32 (adler32-zlib) [ 2288.685535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.785331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.817616] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2289.914837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2289.950704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2292.292925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2292.335105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2292.378792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2292.393460] LNetError: 110485:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2292.429287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2292.462363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2292.770211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2292.839460] LNet: 110506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.845081] LNetError: 110506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2292.848046] LNet: Removed LNI 10.1.2.3@tcp [ 2293.863868] LNet: Removed LNI 192.168.202.10@tcp [ 2294.587036] Key type .llcrypt unregistered [ 2294.598667] Key type ._llcrypt unregistered [ 2308.482617] Key type ._llcrypt registered [ 2308.484804] Key type .llcrypt registered [ 2309.802783] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:40:31 (1787820031) [ 2310.186280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.203799] alg: No test for adler32 (adler32-zlib) [ 2311.173180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.273978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.331254] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2312.548906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2312.660809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg210-client@tcp [ 2319.212515] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:40:41 (1787820041) [ 2319.376513] LNet: 112039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.385327] LNetError: 112039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.411245] LNet: Removed LNI 192.168.202.10@tcp [ 2320.101500] Key type .llcrypt unregistered [ 2320.104992] Key type ._llcrypt unregistered [ 2327.619450] Key type ._llcrypt registered [ 2327.626020] Key type .llcrypt registered [ 2327.961712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.003950] alg: No test for adler32 (adler32-zlib) [ 2329.031335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2329.091949] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2329.134299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2336.753699] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:40:58 (1787820058) [ 2338.236785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2340.126699] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:41:02 (1787820062) [ 2341.570611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2343.528911] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:41:05 (1787820065) [ 2343.853646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.870791] LNet: 113468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.877702] LNetError: 113468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.886883] LNet: Removed LNI 192.168.202.10@tcp [ 2343.905052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.982204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.019839] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2344.667525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2344.734853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2351.577222] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:41:13 (1787820073) [ 2351.664068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.682086] LNet: 114169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.686572] LNetError: 114169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.696815] LNet: Removed LNI 192.168.202.10@tcp [ 2351.707604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.760840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2351.776979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2352.180324] Key type .llcrypt unregistered [ 2352.181985] Key type ._llcrypt unregistered [ 2363.455829] Key type ._llcrypt registered [ 2363.457760] Key type .llcrypt registered [ 2364.380919] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:41:26 (1787820086) [ 2364.877553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2364.897754] alg: No test for adler32 (adler32-zlib) [ 2365.842422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.898864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.921441] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2366.822704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2366.906330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2373.712968] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:41:35 (1787820095) [ 2373.804936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2373.837638] LNet: 115733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.848886] LNetError: 115733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.882554] LNet: Removed LNI 192.168.202.10@tcp [ 2373.958829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.029754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2374.070900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2381.075346] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:41:42 (1787820102) [ 2381.303962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.376513] LNet: 116317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.411806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.496769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2382.051906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2382.087472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2382.168397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2382.794878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2382.885797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2382.958306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.008973] LNet: 116350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.044889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.109948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2383.618661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2383.660732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2383.729870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2384.439803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2384.517125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2391.462801] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:41:53 (1787820113) [ 2391.888706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.920406] LNet: 116951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.943503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.052213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2392.088374] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2393.000152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2393.025851] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2394.171938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2394.244137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2394.458051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.479363] LNetError: 117230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.487320] LNet: Removed LNI 192.168.202.10@tcp [ 2395.490192] LNetError: 117230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.508412] LNet: Removed LNI 192.168.202.10@tcp2 [ 2395.547793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.692602] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2395.716995] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2395.726458] LNet: 117236:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2396.409799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2396.636195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.654405] LNet: 117274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.657755] LNet: 117274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2397.663338] LNet: Removed LNI 192.168.202.10@tcp [ 2397.665764] LNetError: 117274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.675314] LNetError: 117274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2397.688908] LNet: Removed LNI 192.168.202.10@tcp2 [ 2397.731356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.805573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2397.829259] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2398.792496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2398.811747] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2399.845620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2399.890273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2399.947889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2400.186797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.247668] LNet: Removed LNI 192.168.202.10@tcp [ 2402.311037] LNet: Removed LNI 192.168.202.10@tcp2 [ 2402.374843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.504457] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2402.516587] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2402.528440] LNet: 117564:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2403.390557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2403.678112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.707405] LNetError: 117601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.711264] LNetError: 117601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2403.715643] LNet: Removed LNI 192.168.202.10@tcp [ 2404.789765] LNet: Removed LNI 192.168.202.10@tcp2 [ 2404.854423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.934387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.990494] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2405.897988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2405.923508] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2407.286118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2407.362502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2407.621594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2407.656459] LNet: 117881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.668229] LNet: 117881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2408.736293] LNet: Removed LNI 192.168.202.10@tcp [ 2408.749941] LNet: Removed LNI 192.168.202.10@tcp2 [ 2408.804552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.861940] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2408.864653] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2408.868156] LNet: 117887:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2409.262462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2409.578477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2410.659479] LNet: Removed LNI 192.168.202.10@tcp [ 2410.678500] LNet: Removed LNI 192.168.202.10@tcp2 [ 2410.729297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.876386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.940524] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2411.648112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2411.664457] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2413.534739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2413.594821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2413.713134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2413.938769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.954814] LNetError: 118209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.959796] LNetError: 118209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2415.007321] LNet: Removed LNI 192.168.202.10@tcp [ 2416.043331] LNet: Removed LNI 192.168.202.10@tcp2 [ 2416.161808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2416.338286] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2416.357644] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2416.379175] LNet: 118216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2417.216603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2417.603390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.624752] LNet: Removed LNI 192.168.202.10@tcp [ 2418.663816] LNet: Removed LNI 192.168.202.10@tcp2 [ 2418.716935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2418.798744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2418.844379] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2419.751898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2419.808402] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2420.551490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2420.584968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2420.634446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2420.978798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.047193] LNet: Removed LNI 192.168.202.10@tcp [ 2422.063647] LNet: Removed LNI 192.168.202.10@tcp2 [ 2422.137111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.268190] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2422.274609] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2422.282244] LNet: 118555:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.604451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2422.977927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.031358] LNet: Removed LNI 192.168.202.10@tcp [ 2424.040620] LNet: Removed LNI 192.168.202.10@tcp2 [ 2424.060936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.100577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2424.132158] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2424.610394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2424.623132] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2425.216462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2425.241898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2425.265136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2425.305661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2425.564901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2425.580839] LNet: 118909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.583559] LNet: 118909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2426.594785] LNet: Removed LNI 192.168.202.10@tcp [ 2427.621584] LNet: Removed LNI 192.168.202.10@tcp2 [ 2427.705121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.834633] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2427.840788] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2428.605347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2437.018691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2439.107710] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:42:40 (1787820160) [ 2439.208783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2439.266358] LNetError: 119668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.275714] LNetError: 119668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2439.297981] LNet: Removed LNI 192.168.202.10@tcp [ 2440.359586] LNet: Removed LNI 192.168.202.10@tcp2 [ 2440.440804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.648877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2440.669227] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2441.406741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2441.453236] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2442.401758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2442.436890] LNet: Removed LNI 10.1.2.3@tcp [ 2442.448995] LNet: Removed LNI 192.168.202.10@tcp [ 2442.473957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.531868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2442.558185] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2442.561960] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2443.088291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2443.110128] LNet: Removed LNI 192.168.202.10@tcp [ 2443.118783] LNet: Removed LNI 10.1.2.3@tcp [ 2450.466068] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 04:42:51 (1787820171) [ 2450.661897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2450.857536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.991960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2451.038908] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2452.175241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2452.270756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.348858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.424613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.506826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.598225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.659934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.767260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2452.904202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2453.022592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2453.103635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2453.383251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2462.386269] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:43:03 (1787820183) [ 2462.932892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2462.992447] LNet: 121289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.019543] LNet: 121289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2464.119848] LNet: Removed LNI 192.168.202.10@tcp [ 2464.212910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.400498] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2464.493096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2464.501872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2464.517448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2467.290155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2467.354370] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2468.727286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2468.796603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2473.013991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2473.087679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2473.161864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.10@tcp -m PUT -e local_timeout -r 1 [ 2473.244575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2473.314302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2473.319857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2473.693863] LNetError: 121353:0:(lib-move.c:4801:lnet_parse()) 192.168.202.10@tcp, src 192.168.202.10@tcp: Dropping PUT (error -108 looking up sender) [ 2474.726308] LNetError: 121816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2474.731612] LNetError: 121816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2474.738488] LNet: Removed LNI 10.1.2.3@tcp [ 2475.824982] LNet: Removed LNI 192.168.202.10@tcp [ 2476.473995] Key type .llcrypt unregistered [ 2476.477841] Key type ._llcrypt unregistered [ 2488.752972] Key type ._llcrypt registered [ 2488.754568] Key type .llcrypt registered [ 2490.448644] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:43:31 (1787820211) [ 2490.875702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2490.900839] alg: No test for adler32 (adler32-zlib) [ 2491.903819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.419514] LNet: 122669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.047411] Key type .llcrypt unregistered [ 2493.048392] Key type ._llcrypt unregistered [ 2499.787016] Key type ._llcrypt registered [ 2499.788870] Key type .llcrypt registered [ 2500.193500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2500.233172] alg: No test for adler32 (adler32-zlib) [ 2501.277554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2501.383629] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2511.830760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2512.078642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2513.127578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2513.554286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2513.635378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2513.715912] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2513.727083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2513.792262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2513.839194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2513.907263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2513.956721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2513.995779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2514.006548] LNet: 123123:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2514.021815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2514.045198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2514.071480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2514.110654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2514.139326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2514.184745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2514.824092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2515.202556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2515.241936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2515.298520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2515.329712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2515.429885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2515.520659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2515.603880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2515.662597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2515.700207] LNet: 123241:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2515.731576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2515.800295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2515.889144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2515.947470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2516.024434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2516.093902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2517.061625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2517.548249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2517.584548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2517.624909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2517.660770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2517.689965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2517.724336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2517.754543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2517.779698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2517.795874] LNet: 123358:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2517.816914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2517.844411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2517.873275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2517.901166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2517.939576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2517.977195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2519.175958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2519.537723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2519.609102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2519.668678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2519.738215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2519.785581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2519.831591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2519.874286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2519.914393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2519.947879] LNet: 123475:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2519.987629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2520.055281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2520.142679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2520.242051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2520.315658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2520.424359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2521.000996] LNet: 123525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.006983] LNetError: 123525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.029681] LNet: Removed LNI 192.168.202.10@tcp [ 2521.620768] Key type .llcrypt unregistered [ 2521.625448] Key type ._llcrypt unregistered [ 2541.470685] Key type ._llcrypt registered [ 2541.472217] Key type .llcrypt registered [ 2543.313649] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:44:24 (1787820264) [ 2543.531468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.539639] alg: No test for adler32 (adler32-zlib) [ 2544.462250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.857701] LNet: 124414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.565822] Key type .llcrypt unregistered [ 2545.571657] Key type ._llcrypt unregistered [ 2553.423247] Key type ._llcrypt registered [ 2553.427879] Key type .llcrypt registered [ 2553.899143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2553.969915] alg: No test for adler32 (adler32-zlib) [ 2554.994617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2555.059509] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2564.316796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2575.137834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2575.203631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2575.245669] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2576.431235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2577.098484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2577.519491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2577.565454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2577.631476] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2577.631752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2577.700293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2577.787866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2577.834979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2577.902449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2577.958442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2577.986432] LNet: 125071:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2578.022028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2578.078218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2578.134851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2578.180293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2578.220328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2578.268780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2579.195961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2579.570626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2579.609798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2579.660887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2579.697487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2579.746395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2579.786878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2579.832113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2579.879611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2579.909608] LNet: 125188:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2579.946392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2579.987815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2580.048662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2580.109074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2580.178141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2580.232776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2580.874120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2581.160512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2581.232672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2581.307881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2581.380662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2581.455933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2581.547992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2581.619660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2581.716142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2581.758959] LNet: 125305:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2581.823731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2581.909268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2581.992098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2582.039882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2582.072297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2582.104386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2583.274379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2583.561498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2583.609948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2583.655759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2583.692565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2583.733919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2583.769474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2583.805080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2583.848920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2583.896443] LNet: 125423:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2583.947785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2584.006840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2584.075761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2584.148672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2584.212648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2584.272562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2590.371783] LNet: 125508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.385573] LNetError: 125508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.394720] LNet: Removed LNI 192.168.202.10@tcp [ 2590.404923] LNet: Removed LNI 192.168.202.10@tcp1 [ 2590.886323] Key type .llcrypt unregistered [ 2590.895534] Key type ._llcrypt unregistered [ 2612.354149] Key type ._llcrypt registered [ 2612.359636] Key type .llcrypt registered [ 2613.905460] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 04:45:35 (1787820335) [ 2614.505671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2614.527450] alg: No test for adler32 (adler32-zlib) [ 2615.423842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.509703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2615.548843] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2616.510589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2616.533821] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2617.861254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2617.994759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2619.039183] LNet: Removed LNI 192.168.202.10@tcp [ 2628.624507] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 04:45:49 (1787820349) [ 2630.999831] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2633.401321] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:45:55 (1787820355) [ 2643.659031] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:46:05 (1787820365) [ 2652.705361] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:46:14 (1787820374) [ 2660.478833] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:46:22 (1787820382) [ 2669.771248] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:46:31 (1787820391) [ 2678.419569] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:46:39 (1787820399) [ 2687.199106] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:46:49 (1787820409) [ 2697.373448] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:46:58 (1787820418) [ 2706.265509] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:47:07 (1787820427) [ 2715.673772] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:47:17 (1787820437) [ 2724.700327] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:47:26 (1787820446) [ 2732.490096] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:47:34 (1787820454) [ 2741.085239] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:47:42 (1787820462) [ 2748.946793] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:47:50 (1787820470) [ 2757.731202] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:47:59 (1787820479) [ 2766.399221] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:48:08 (1787820488) [ 2775.014223] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:48:16 (1787820496) [ 2782.692250] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:48:24 (1787820504) [ 2782.901991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2782.944863] LNet: 137204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2782.959129] LNetError: 137204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2782.987391] LNet: Removed LNI 192.168.202.10@tcp2 [ 2783.043147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2783.218561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2783.261305] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2784.813553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2784.855513] LNet: 137349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2784.864169] LNetError: 137349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2784.884369] LNet: Removed LNI 192.168.202.10@tcp [ 2784.909390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2784.986308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2785.014430] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2791.955477] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:48:33 (1787820513) [ 2800.021531] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:48:41 (1787820521) [ 2808.150391] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:48:50 (1787820530) [ 2816.796407] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:48:58 (1787820538) [ 2823.591543] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:49:05 (1787820545) [ 2832.143263] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:49:13 (1787820553) [ 2839.543746] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:49:21 (1787820561) [ 2846.698033] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:49:28 (1787820568) [ 2856.193498] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:49:37 (1787820577) [ 2864.691913] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:49:46 (1787820586) [ 2873.235350] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:49:54 (1787820594) [ 2881.766170] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:50:03 (1787820603) [ 2889.817181] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2603 sec ========== 04:50:11 (1787820611) [ 2890.199896] LNet: 144815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2890.203624] LNetError: 144815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.222649] LNet: Removed LNI 192.168.202.10@tcp [ 2890.984991] Key type .llcrypt unregistered [ 2890.986526] Key type ._llcrypt unregistered [ 2906.717437] Key type ._llcrypt registered [ 2906.734088] Key type .llcrypt registered [ 2907.569750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2907.598987] alg: No test for adler32 (adler32-zlib) [ 2908.809419] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2909.137224] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2910.904260] Key type lgssc registered [ 2912.243215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2982.765849] Lustre: Mounted lustre-client - version 2.17.57_80_gdb9f866 [ 2988.653512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2998.708519] Lustre: Unmounted lustre-client [ 3030.955740] Key type lgssc unregistered [ 3031.248647] LNet: 147780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3031.253892] LNetError: 147780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3031.273509] LNet: Removed LNI 192.168.202.10@tcp [ 3032.127189] Key type .llcrypt unregistered [ 3032.130530] Key type ._llcrypt unregistered