[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 441820523 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003374] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008312] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.012157] LSM: Security Framework initializing [ 0.013055] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.016068] *** VALIDATE selinux *** [ 0.024080] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029507] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032039] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034047] *** VALIDATE tmpfs *** [ 0.035463] *** VALIDATE proc *** [ 0.037165] *** VALIDATE cgroup *** [ 0.038012] *** VALIDATE cgroup2 *** [ 0.040109] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043031] Spectre V2 : User space: Vulnerable [ 0.044011] Speculative Store Bypass: Vulnerable [ 0.047186] debug: unmapping init [mem 0xffffffffafe59000-0xffffffffafe60fff] [ 0.049958] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050587] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051025] ... version: 2 [ 0.052013] ... bit width: 48 [ 0.053013] ... generic registers: 4 [ 0.054013] ... value mask: 0000ffffffffffff [ 0.055015] ... max period: 00007fffffffffff [ 0.056015] ... fixed-purpose events: 3 [ 0.057013] ... event mask: 000000070000000f [ 0.058287] rcu: Hierarchical SRCU implementation. [ 0.060366] smp: Bringing up secondary CPUs ... [ 0.061538] x86: Booting SMP configuration: [ 0.062026] .... node #0, CPUs: #1 #2 #3 [ 0.065650] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151041] node 0 deferred pages initialised in 80ms [ 0.154130] devtmpfs: initialized [ 0.155229] x86/mm: Memory block size: 128MB [ 0.157821] gcov: version magic: 0x41383552 [ 0.158575] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164338] pinctrl core: initialized pinctrl subsystem [ 0.166160] [ 0.166667] ************************************************************* [ 0.169013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171010] ** ** [ 0.173011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175012] ** ** [ 0.177011] ** This means that this kernel is built to expose internal ** [ 0.180012] ** IOMMU data structures, which may compromise security on ** [ 0.182012] ** your system. ** [ 0.184011] ** ** [ 0.186012] ** If you see this message and you are not debugging the ** [ 0.188012] ** kernel, report this immediately to your vendor! ** [ 0.190013] ** ** [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195014] ************************************************************* [ 0.197645] NET: Registered protocol family 16 [ 0.199392] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208060] cpuidle: using governor menu [ 0.209866] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212446] PCI: Using configuration type 1 for base access [ 0.214120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225186] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231083] cryptd: max_cpu_qlen set to 1000 [ 0.234249] ACPI: Added _OSI(Module Device) [ 0.236016] ACPI: Added _OSI(Processor Device) [ 0.237014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244254] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250344] ACPI: Interpreter enabled [ 0.252060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253013] ACPI: Using IOAPIC for interrupt routing [ 0.255107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258354] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269341] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281394] acpiphp: Slot [2] registered [ 0.283079] acpiphp: Slot [5] registered [ 0.284103] acpiphp: Slot [6] registered [ 0.285154] acpiphp: Slot [3] registered [ 0.287099] acpiphp: Slot [4] registered [ 0.288088] acpiphp: Slot [7] registered [ 0.289075] acpiphp: Slot [8] registered [ 0.291078] acpiphp: Slot [9] registered [ 0.292107] acpiphp: Slot [10] registered [ 0.293065] acpiphp: Slot [11] registered [ 0.294160] acpiphp: Slot [12] registered [ 0.296121] acpiphp: Slot [13] registered [ 0.297102] acpiphp: Slot [14] registered [ 0.299103] acpiphp: Slot [15] registered [ 0.300094] acpiphp: Slot [16] registered [ 0.302122] acpiphp: Slot [17] registered [ 0.303097] acpiphp: Slot [18] registered [ 0.305092] acpiphp: Slot [19] registered [ 0.306060] acpiphp: Slot [20] registered [ 0.307000] acpiphp: Slot [21] registered [ 0.307000] acpiphp: Slot [22] registered [ 0.309108] acpiphp: Slot [23] registered [ 0.310099] acpiphp: Slot [24] registered [ 0.312114] acpiphp: Slot [25] registered [ 0.313081] acpiphp: Slot [26] registered [ 0.314094] acpiphp: Slot [27] registered [ 0.316094] acpiphp: Slot [28] registered [ 0.317087] acpiphp: Slot [29] registered [ 0.318112] acpiphp: Slot [30] registered [ 0.320104] acpiphp: Slot [31] registered [ 0.321048] PCI host bridge to bus 0000:00 [ 0.323024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342221] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.354055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364459] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367784] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372702] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.387015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.393848] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399090] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436638] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.468016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480754] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.482349] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.484310] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.486326] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.488209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.492130] iommu: Default domain type: Passthrough [ 0.493365] SCSI subsystem initialized [ 0.495137] ACPI: bus type USB registered [ 0.496106] usbcore: registered new interface driver usbfs [ 0.498087] usbcore: registered new interface driver hub [ 0.500074] usbcore: registered new device driver usb [ 0.502151] pps_core: LinuxPPS API ver. 1 registered [ 0.504041] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.506044] PTP clock support registered [ 0.508085] EDAC MC: Ver: 3.0.0 [ 0.510269] PCI: Using ACPI for IRQ routing [ 0.511521] NetLabel: Initializing [ 0.512013] NetLabel: domain hash size = 128 [ 0.513009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.515075] NetLabel: unlabeled traffic allowed by default [ 0.517127] vgaarb: loaded [ 0.519259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.521011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.527402] clocksource: Switched to clocksource kvm-clock [ 0.630682] VFS: Disk quotas dquot_6.6.0 [ 0.632072] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.634474] *** VALIDATE ramfs *** [ 0.635694] *** VALIDATE hugetlbfs *** [ 0.636996] pnp: PnP ACPI init [ 0.639107] pnp: PnP ACPI: found 6 devices [ 0.655206] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.658342] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.660507] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.662635] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.664903] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.667206] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.669964] NET: Registered protocol family 2 [ 0.672370] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.676578] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.680106] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.685553] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.689157] TCP: Hash tables configured (established 65536 bind 65536) [ 0.691993] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.695179] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.698243] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.701456] NET: Registered protocol family 1 [ 0.704755] RPC: Registered named UNIX socket transport module. [ 0.706928] RPC: Registered udp transport module. [ 0.708686] RPC: Registered tcp transport module. [ 0.710494] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.712913] NET: Registered protocol family 44 [ 0.714482] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.716564] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.718468] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.720565] PCI: CLS 0 bytes, default 64 [ 0.722174] Unpacking initramfs... [ 2.115754] debug: unmapping init [mem 0xffff9dc97cc64000-0xffff9dc97ffcffff] [ 2.119673] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.121983] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.124995] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.615235] Initialise system trusted keyrings [ 2.616971] Key type blacklist registered [ 2.619097] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.628250] zbud: loaded [ 2.631166] *** VALIDATE nfs *** [ 2.632495] *** VALIDATE nfs4 *** [ 2.633942] pstore: using deflate compression [ 2.637169] Platform Keyring initialized [ 2.744648] NET: Registered protocol family 38 [ 2.746522] Key type asymmetric registered [ 2.748267] Asymmetric key parser 'x509' registered [ 2.750092] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.752977] io scheduler mq-deadline registered [ 2.754491] io scheduler kyber registered [ 2.755956] io scheduler bfq registered [ 2.758072] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.761066] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.763875] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.766712] ACPI: Power Button [PWRF] [ 2.771791] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.777964] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.801072] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.833407] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.865448] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.870036] Non-volatile memory driver v1.3 [ 2.871572] Linux agpgart interface v0.103 [ 2.900904] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.903736] vda: detected capacity change from 0 to 74706944 [ 2.919860] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.922931] vdb: detected capacity change from 0 to 1073741824 [ 2.929358] libphy: Fixed MDIO Bus: probed [ 2.942284] usbcore: registered new interface driver usbserial_generic [ 2.944216] usbserial: USB Serial support registered for generic [ 2.946280] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.949718] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.951230] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.953851] mousedev: PS/2 mouse device common for all mice [ 2.956871] rtc_cmos 00:05: RTC can wake from S4 [ 2.959944] rtc_cmos 00:05: registered as rtc0 [ 2.961667] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.965245] intel_pstate: CPU model not supported [ 2.968217] hid: raw HID events driver (C) Jiri Kosina [ 2.969072] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.972960] usbcore: registered new interface driver usbhid [ 2.978783] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.979307] usbhid: USB HID core driver [ 2.983760] drop_monitor: Initializing network drop monitor service [ 2.983811] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.986067] Initializing XFRM netlink socket [ 2.990866] NET: Registered protocol family 10 [ 2.993309] Segment Routing with IPv6 [ 2.994636] NET: Registered protocol family 17 [ 2.997269] mpls_gso: MPLS GSO support [ 3.002501] RAS: Correctable Errors collector initialized. [ 3.004471] AVX version of gcm_enc/dec engaged. [ 3.006475] AES CTR mode by8 optimization enabled [ 3.073037] sched_clock: Marking stable (3073009590, 0)->(3964855539, -891845949) [ 3.076250] registered taskstats version 1 [ 3.077885] Loading compiled-in X.509 certificates [ 3.080399] zswap: loaded using pool lzo/zbud [ 3.104987] Key type big_key registered [ 3.115776] Key type encrypted registered [ 3.117195] ima: No TPM chip found, activating TPM-bypass! [ 3.119261] ima: Allocated hash algorithm: sha1 [ 3.120809] ima: No architecture policies found [ 3.122674] evm: Initialising EVM extended attributes: [ 3.124534] evm: security.selinux [ 3.125739] evm: security.ima [ 3.126870] evm: security.capability [ 3.128100] evm: HMAC attrs: 0x1 [ 3.130358] rtc_cmos 00:05: setting system clock to 2026-08-14 18:23:43 UTC (1786731823) [ 3.136986] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 3.139935] debug: unmapping init [mem 0xffffffffafb82000-0xffffffffafe58fff] [ 3.148258] Write protecting the kernel read-only data: 28672k [ 3.151310] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.154130] debug: unmapping init [mem 0xffffffffaeb14000-0xffffffffaebfffff] [ 3.184116] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.190789] systemd[1]: Detected virtualization kvm. [ 3.192727] systemd[1]: Detected architecture x86-64. [ 3.194604] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.220632] systemd[1]: No hostname configured. [ 3.222451] systemd[1]: Set hostname to . [ 3.223878] random: systemd: uninitialized urandom read (16 bytes read) [ 3.225308] systemd[1]: Initializing machine ID from random generator. [ 3.270027] random: ln: uninitialized urandom read (6 bytes read) [ 3.359231] random: systemd: uninitialized urandom read (16 bytes read) [ 3.361892] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.366914] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.372579] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.036603] device-mapper: uevent: version 1.0.3 [ 4.038551] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook.[ 4.440404] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.818935] virtio_net virtio0 ens2: renamed from eth0 [ 4.883351] scsi host0: ata_piix [ 5.011447] scsi host1: ata_piix [ 5.015982] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.017837] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.780947] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.637756] random: crng init done [ 9.639096] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.175093] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.392385] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.648562] SELinux: Disabled at runtime. [ 11.715823] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.722919] systemd[1]: Detected virtualization kvm. [ 11.724694] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.240518] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.244303] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.250451] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.254542] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.258058] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.267204] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.270750] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK [0[ 12.446337] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.748681] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.042310] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.072133] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.169608] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.187728] EDAC sbridge: Ver: 1.1.2 [ 14.195901] Key type dns_resolver registered [ 14.496868] NFS: Registering the id_resolver key type [ 14.498899] Key type id_resolver registered [ 14.500428] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg659-client login: [ 45.469551] libcfs: loading out-of-tree module taints kernel. [ 45.599632] Key type ._llcrypt registered [ 45.601572] Key type .llcrypt registered [ 45.845190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.854438] alg: No test for adler32 (adler32-zlib) [ 46.952471] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 47.343176] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 48.984210] Key type lgssc registered [ 51.315057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.147886] hrtimer: interrupt took 3877899 ns [ 176.851409] Lustre: Mounted lustre-client [ 181.937442] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 199.525542] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing check_logdir /tmp/testlogs/ [ 202.721489] Lustre: lustre-OST0000-osc-ffff9dc9d848e800: disconnect after 24s idle [ 204.502545] Lustre: DEBUG MARKER: oleg659-client.virtnet: executing yml_node [ 209.310618] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 212.277496] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 215.223340] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 217.837750] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 14:27:15 EDT 2026 [ 233.971460] Lustre: Unmounted lustre-client [ 262.610961] Key type lgssc unregistered [ 262.984922] LNet: 8401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 262.999296] LNetError: 8401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 263.027691] LNet: Removed LNI 192.168.206.59@tcp [ 263.592911] Key type .llcrypt unregistered [ 263.598361] Key type ._llcrypt unregistered [ 288.804712] Key type ._llcrypt registered [ 288.817593] Key type .llcrypt registered [ 289.116922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 289.134790] alg: No test for adler32 (adler32-zlib) [ 290.140991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 290.222159] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 290.256898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 294.154642] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 294.407670] LNet: 9301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.412712] LNetError: 9301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 294.431057] LNet: Removed LNI 192.168.206.59@tcp [ 295.134556] Key type .llcrypt unregistered [ 295.137406] Key type ._llcrypt unregistered [ 301.387753] Key type ._llcrypt registered [ 301.391159] Key type .llcrypt registered [ 303.056219] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:28:41 (1786732121) [ 303.344816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 303.353234] alg: No test for adler32 (adler32-zlib) [ 304.269662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.353830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.388436] LNet: 9712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.418977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.488763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 304.533141] LNet: 9719:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 312.113420] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:28:50 (1786732130) [ 312.220829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.247693] LNet: 10291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.285205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.466225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 320.126779] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:28:58 (1786732138) [ 320.243412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.312500] LNet: 10875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.354866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.488040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 329.165128] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:29:07 (1786732147) [ 329.210163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.229593] LNet: 11460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.260733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.366607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 336.469515] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:29:15 (1786732155) [ 336.652173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.672956] LNet: 12052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.691627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.779322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 337.281531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 343.971728] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:29:22 (1786732162) [ 344.115675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.160438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.220755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 344.615611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 344.944514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 350.782437] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:29:29 (1786732169) [ 350.891083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.950581] LNet: 13234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.965175] LNet: 13234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 351.016426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.106624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 357.432729] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:29:36 (1786732176) [ 357.734783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.798987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.879435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 357.924992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 358.328989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 358.378421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 358.908578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 358.975627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.396478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 359.449521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.898278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 359.946280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.352392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 360.399882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 360.800463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 360.857477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.291683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 361.328822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 361.707774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 361.754519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.159956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 362.183972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 362.543440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 362.582723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 362.880786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 362.914389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.189029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 363.250992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 369.164482] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:29:48 (1786732188) [ 369.459955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.480817] LNet: 14507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.486149] LNet: 14507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.513469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.606297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 369.653886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 376.511391] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:29:55 (1786732195) [ 376.612077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.651829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.728450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 376.753890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 383.295719] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:30:02 (1786732202) [ 383.418484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.510267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.590609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 383.616681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 391.284064] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:30:09 (1786732209) [ 391.579681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.662288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.825405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 391.932900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 400.064289] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:30:18 (1786732218) [ 400.334368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.400217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.489766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 400.547222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 408.344522] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:30:26 (1786732226) [ 408.480481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.508295] LNet: 17435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.511293] LNet: 17435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 408.529202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.622978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 408.660958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 416.345115] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:30:35 (1786732235) [ 416.462408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.529762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.659932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.030717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 424.128302] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:30:43 (1786732243) [ 424.253778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.310643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.490840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.659569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.206837] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:30:50 (1786732250) [ 431.358038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.412498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.622812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 431.892841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.969860] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:30:58 (1786732258) [ 440.283537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.387233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.833823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.076418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.013974] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:31:06 (1786732266) [ 448.161448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.306827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.510855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.570858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.241774] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:31:15 (1786732275) [ 456.722984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.780526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.849963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.933575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.955162] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:31:24 (1786732284) [ 465.147912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.227904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.387368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 472.920272] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:31:31 (1786732291) [ 473.160065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.197967] LNet: 22294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.204829] LNet: 22294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 473.237298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.344486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 480.568733] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:31:39 (1786732299) [ 480.688954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.748749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.824861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 481.158293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 490.216684] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:31:48 (1786732308) [ 490.343643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.420483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.671233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 490.734484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 500.124956] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:31:58 (1786732318) [ 500.254430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.300074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.420691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 500.460516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 508.860119] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:32:07 (1786732327) [ 508.950885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.032903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.201405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 509.258557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 518.239613] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:32:16 (1786732336) [ 518.341268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.386033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.548689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 518.608327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 527.428689] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:32:25 (1786732345) [ 527.558934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.636812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.759852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 527.915352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 527.959989] LNetError: 25828:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 528.061270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 528.206230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 535.668442] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:32:34 (1786732354) [ 535.890526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.934732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.069075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.eLWIe676w5 [ 536.234435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.eLWIe676w5 [ 536.333591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.eLWIe676w5 [ 543.167690] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:32:42 (1786732362) [ 543.474560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.558706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.648627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 543.729352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 543.808961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.964264] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:32:48 (1786732368) [ 550.128084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.157908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.217961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 550.241566] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 551.852542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 552.994597] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 554.872199] Key type lgssc registered [ 556.280376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 559.666676] LNetError: 27590:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.59@tcp99 has no available nets [ 562.281148] Key type lgssc unregistered [ 562.603291] LNetError: 28698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.627652] LNet: Removed LNI 192.168.206.59@tcp [ 564.225175] Key type .llcrypt unregistered [ 564.226990] Key type ._llcrypt unregistered [ 570.005851] Key type ._llcrypt registered [ 570.009512] Key type .llcrypt registered [ 571.373302] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:33:10 (1786732390) [ 571.746690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 571.760635] alg: No test for adler32 (adler32-zlib) [ 572.665223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.752437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 575.598429] LNet: Message forwarding will be enabled in 2s [ 577.632123] LNet: Message forwarding enabled [ 577.951426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 577.974589] LNet: Message forwarding will stop in 2s to 6s [ 582.492592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 583.582714] LNet: Message forwarding disabled [ 584.962793] LNet: Message forwarding will be enabled in 2s [ 588.705438] LNet: Message forwarding enabled [ 591.568715] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:33:30 (1786732410) [ 592.018923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.062828] LNet: 30096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.581082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.706559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 601.126143] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:33:39 (1786732419) [ 601.274912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.299857] LNet: 30672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.326722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.426742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 601.466272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 601.514744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 601.560229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 601.594173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 601.659282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 601.702783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 601.757517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 601.803945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 601.859469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 601.887498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 601.928071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 601.963421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 601.991606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 602.018211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 602.060471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 602.095167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 602.129173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 602.167280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 602.186902] LNetError: 30720:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 609.278662] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:33:47 (1786732427) [ 609.394756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.429193] LNet: 31285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.459246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.572980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.441081] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:33:57 (1786732437) [ 620.379716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg659-server [ 622.525793] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:34:01 (1786732441) [ 622.611910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.632547] LNet: 32214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.670132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.746492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.804982] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 623.788249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[60-64]@tcp [ 623.812832] LNet: 32356:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 626.921289] LNetError: 32357:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.63@tcp has gone from up to down [ 630.049627] LNetError: 32233:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.64@tcp has gone from up to down [ 630.068278] LNetError: 32233:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 632.587754] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:34:11 (1786732451) [ 634.971660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg659-server [ 636.986187] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:34:15 (1786732455) [ 637.264374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.176425] LNetError: 32224:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.206.64@tcp [ 640.416905] LNet: 33293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.438682] LNetError: 33293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.483930] LNet: Removed LNI 192.168.206.59@tcp [ 640.543918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.620879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.668500] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 641.763296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[60-65/2]@tcp [ 641.786953] LNet: 33410:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.799352] LNet: 33410:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 641.813993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[60-65/2]@tcp [ 648.478390] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:34:27 (1786732467) [ 648.640974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.664799] LNet: 34002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.671885] LNetError: 34002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.682031] LNet: Removed LNI 192.168.206.59@tcp [ 648.702857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.777654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 648.844363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.873238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.966926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.040803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.122527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.199915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 649.339310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.411551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.470383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.543579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.600723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.660802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 649.754827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.793316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.911894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.978700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.044989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.132735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 650.229643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.280064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.366882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 650.493198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.517256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.556699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 650.618257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.654988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.726917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 650.801541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.851804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.902975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 650.948996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.993339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.118673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.520650] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:34:36 (1786732476) [ 659.219846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg659-server [ 660.797856] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:34:40 (1786732480) [ 662.217221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg659-server [ 664.017287] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:34:43 (1786732483) [ 664.857673] Key type .llcrypt unregistered [ 664.861709] Key type ._llcrypt unregistered [ 670.365683] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 670.614173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 670.617103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 670.620219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 673.005826] Key type ._llcrypt registered [ 673.008876] Key type .llcrypt registered [ 673.189704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 673.221207] alg: No test for adler32 (adler32-zlib) [ 674.167665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.236307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.266093] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 674.967796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 674.998453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 677.195247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 677.215387] LNet: Removed LNI 10.1.2.3@tcp [ 677.332391] LNet: 36126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.341790] LNetError: 36126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.367608] LNet: Removed LNI 192.168.206.59@tcp [ 677.907503] Key type .llcrypt unregistered [ 677.911917] Key type ._llcrypt unregistered [ 690.082594] Key type ._llcrypt registered [ 690.086072] Key type .llcrypt registered [ 691.911954] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:35:10 (1786732510) [ 692.551432] Key type .llcrypt unregistered [ 692.554258] Key type ._llcrypt unregistered [ 700.517407] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.851683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.864069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.890919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.632180] Key type ._llcrypt registered [ 703.643725] Key type .llcrypt registered [ 703.919473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.930939] alg: No test for adler32 (adler32-zlib) [ 704.922189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.019175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.076965] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 705.735737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.763393] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.173367] LNet: 37619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.184269] LNetError: 37619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 709.200559] LNet: Removed LNI 10.1.2.3@tcp [ 709.225610] LNet: Removed LNI 192.168.206.59@tcp [ 710.160679] Key type .llcrypt unregistered [ 710.164033] Key type ._llcrypt unregistered [ 722.862544] Key type ._llcrypt registered [ 722.864588] Key type .llcrypt registered [ 723.970271] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:35:43 (1786732543) [ 724.750683] Key type .llcrypt unregistered [ 724.752830] Key type ._llcrypt unregistered [ 730.673226] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 730.820963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 730.833195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 730.846783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.409340] Key type ._llcrypt registered [ 733.411655] Key type .llcrypt registered [ 733.536174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.545449] alg: No test for adler32 (adler32-zlib) [ 734.427649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.479502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 734.507627] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 736.183964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 736.201955] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.690806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 737.722873] LNet: Removed LNI 10.1.2.3@tcp [ 737.739429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 737.765383] LNet: Removed LNI 10.1.2.31@tcp [ 737.780205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 737.805398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.301358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 738.321118] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 739.255766] LNet: 39535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.260414] LNetError: 39535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 739.269132] LNet: Removed LNI 10.1.2.31@tcp [ 739.288154] LNet: Removed LNI 10.1.2.3@tcp [ 739.816652] Key type .llcrypt unregistered [ 739.819651] Key type ._llcrypt unregistered [ 750.034662] Key type ._llcrypt registered [ 750.038986] Key type .llcrypt registered [ 751.253611] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:36:10 (1786732570) [ 751.716420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.735510] alg: No test for adler32 (adler32-zlib) [ 752.699511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.757286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.781382] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 753.532864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.568568] LNet: 40488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.574989] LNetError: 40488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.590963] LNet: Removed LNI 192.168.206.59@tcp [ 753.641394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.733882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.59@tcp [ 753.753043] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 754.763296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.59@tcp [ 754.788340] LNet: Removed LNI 192.168.206.59@tcp [ 754.838435] LNet: 40623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.363669] Key type .llcrypt unregistered [ 755.368530] Key type ._llcrypt unregistered [ 767.009927] Key type ._llcrypt registered [ 767.011650] Key type .llcrypt registered [ 768.546965] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:36:27 (1786732587) [ 770.881170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg659-server [ 773.255736] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:36:31 (1786732591) [ 774.026516] Key type .llcrypt unregistered [ 774.031814] Key type ._llcrypt unregistered [ 780.348179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.502063] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.509735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.529344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.085982] Key type ._llcrypt registered [ 783.092799] Key type .llcrypt registered [ 783.271482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.292853] alg: No test for adler32 (adler32-zlib) [ 784.196486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.254338] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 784.821270] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 786.825308] LNet: 42459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.829692] LNetError: 42459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.832970] LNet: Removed LNI 192.168.206.59@tcp1 [ 786.839932] LNet: Removed LNI 10.1.2.3@tcp2 [ 787.633820] Key type .llcrypt unregistered [ 787.639362] Key type ._llcrypt unregistered [ 798.602356] Key type ._llcrypt registered [ 798.607090] Key type .llcrypt registered [ 800.128296] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:36:58 (1786732618) [ 800.649755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 800.664879] alg: No test for adler32 (adler32-zlib) [ 801.595178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.640418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.656901] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 802.470842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.486888] LNet: 43428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.495697] LNetError: 43428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.512664] LNet: Removed LNI 192.168.206.59@tcp [ 802.528344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.595273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 802.637596] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 809.881391] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:37:08 (1786732628) [ 810.076226] LNet: 44011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.089925] LNetError: 44011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.118613] LNet: Removed LNI 192.168.206.59@tcp [ 811.240498] Key type .llcrypt unregistered [ 811.241969] Key type ._llcrypt unregistered [ 817.654779] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.733860] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.742284] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.747327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.310335] Key type ._llcrypt registered [ 820.313787] Key type .llcrypt registered [ 820.506582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.535822] alg: No test for adler32 (adler32-zlib) [ 821.451903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.530593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.563558] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 822.364893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.396223] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.703269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.727547] LNet: 44759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.742377] LNetError: 44759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.752169] LNet: Removed LNI 10.1.2.3@tcp [ 824.760880] LNet: Removed LNI 192.168.206.59@tcp [ 824.797903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.856532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.898097] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 824.909401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.314175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.337699] LNet: 44786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.344719] LNetError: 44786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 825.351739] LNetError: 44786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 825.357487] LNet: Removed LNI 10.1.2.3@tcp [ 825.369934] LNet: Removed LNI 192.168.206.59@tcp [ 825.394739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.492496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 825.521107] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 825.525467] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.789731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 825.808419] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 826.505417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.527139] LNet: 44949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.536895] LNetError: 44949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 826.542851] LNetError: 44949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 826.549340] LNet: Removed LNI 10.1.2.3@tcp [ 826.551688] LNet: Removed LNI 192.168.206.59@tcp [ 826.560724] LNet: Removed LNI 192.168.206.59@tcp2 [ 826.580969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.638561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 826.657973] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 826.663032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.670833] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 833.816977] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:37:32 (1786732652) [ 833.918290] LNet: 45539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.932168] LNetError: 45539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 833.937113] LNetError: 45539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 833.945414] LNet: Removed LNI 10.1.2.3@tcp [ 833.946828] LNet: Removed LNI 192.168.206.59@tcp [ 833.970152] LNet: Removed LNI 192.168.206.59@tcp2 [ 834.749282] Key type .llcrypt unregistered [ 834.755735] Key type ._llcrypt unregistered [ 840.747020] Key type ._llcrypt registered [ 840.749799] Key type .llcrypt registered [ 840.968838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.985180] alg: No test for adler32 (adler32-zlib) [ 841.916753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.986362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.020792] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 842.994519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.015866] LNet: 45942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.023257] LNetError: 45942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.035401] LNet: Removed LNI 192.168.206.59@tcp [ 843.059547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.107129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 843.125756] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 843.549746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.585437] LNet: 45972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.593832] LNetError: 45972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.621107] LNet: Removed LNI 192.168.206.59@tcp [ 843.647448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.757828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 843.816804] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 844.477396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.509908] LNet: Removed LNI 192.168.206.59@tcp [ 844.541746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.603926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.620790] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 845.050444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 845.162032] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 845.170075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 845.175403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 847.577442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.599660] LNet: 46075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.605616] LNet: 46075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 847.612783] LNetError: 46075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.619650] LNetError: 46075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 847.640415] LNet: Removed LNI 192.168.206.59@tcp [ 847.664412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.721399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 847.760650] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 855.384177] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:37:54 (1786732674) [ 855.520532] LNet: 46668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.529975] LNetError: 46668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.550390] LNet: Removed LNI 192.168.206.59@tcp [ 856.280068] Key type .llcrypt unregistered [ 856.283826] Key type ._llcrypt unregistered [ 863.158708] Key type ._llcrypt registered [ 863.160962] Key type .llcrypt registered [ 863.414889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.434929] alg: No test for adler32 (adler32-zlib) [ 864.389963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.482859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.513463] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 865.585708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.621717] LNet: 47071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.632544] LNetError: 47071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.651307] LNet: Removed LNI 192.168.206.59@tcp [ 865.690097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.780534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 865.827836] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 872.100316] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:38:11 (1786732691) [ 872.360024] LNet: 47663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.375966] LNetError: 47663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.393066] LNet: Removed LNI 192.168.206.59@tcp [ 872.971686] Key type .llcrypt unregistered [ 872.974324] Key type ._llcrypt unregistered [ 879.858453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 879.935465] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 879.945507] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 879.962079] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 882.841579] Key type ._llcrypt registered [ 882.844208] Key type .llcrypt registered [ 883.005592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.020148] alg: No test for adler32 (adler32-zlib) [ 883.926662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.976285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.003308] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 884.809275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.827533] LNet: 48113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.842848] LNetError: 48113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.862644] LNet: Removed LNI 192.168.206.59@tcp [ 884.908738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.003591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 885.034701] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 891.417911] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:38:30 (1786732710) [ 891.684509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.708334] LNet: 48711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.714251] LNetError: 48711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.733298] LNet: Removed LNI 192.168.206.59@tcp [ 891.768776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.866345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 898.884565] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:38:37 (1786732717) [ 898.988675] LNet: 49286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.862684] Key type .llcrypt unregistered [ 899.864572] Key type ._llcrypt unregistered [ 906.437887] Key type ._llcrypt registered [ 906.440251] Key type .llcrypt registered [ 906.624479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.658646] alg: No test for adler32 (adler32-zlib) [ 907.614178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.675893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.714319] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 908.610144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.639245] LNet: 49694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.653503] LNetError: 49694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.669655] LNet: Removed LNI 192.168.206.59@tcp [ 908.701843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.801725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 908.858485] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 915.135686] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:38:54 (1786732734) [ 915.318665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.337697] LNet: 50288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.346564] LNetError: 50288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.371744] LNet: Removed LNI 192.168.206.59@tcp [ 915.387548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.457833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 920.669275] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:38:59 (1786732739) [ 921.819825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 923.585807] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:39:02 (1786732742) [ 924.766406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 926.806681] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:39:05 (1786732745) [ 928.380665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 929.772320] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:39:09 (1786732749) [ 931.325933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 933.000753] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:39:12 (1786732752) [ 934.874556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 936.684053] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:39:15 (1786732755) [ 936.841028] LNet: 52332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.606132] Key type .llcrypt unregistered [ 937.610440] Key type ._llcrypt unregistered [ 943.619526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.819110] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.828869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.848328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.376831] Key type ._llcrypt registered [ 946.379222] Key type .llcrypt registered [ 946.607403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.623175] alg: No test for adler32 (adler32-zlib) [ 947.578677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.683683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.713289] LNet: Added LNI 192.168.206.59@tcp [8/512/0/180] [ 947.718992] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 947.720990] LNet: 52661:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 947.787782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.807633] LNet: 52680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.814977] LNetError: 52680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.826850] LNet: Removed LNI 10.1.2.3@tcp [ 947.830240] LNetError: 52680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.851817] LNet: Removed LNI 192.168.206.59@tcp [ 947.887079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.968964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.005883] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 948.060253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.074128] LNet: 52704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.078615] LNetError: 52704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.096581] LNet: Removed LNI 192.168.206.59@tcp [ 948.135192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.244094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.293339] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 948.304991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 948.306906] LNet: 52712:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.417433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.431933] LNet: 52730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.436526] LNetError: 52730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.440670] LNet: Removed LNI 10.1.2.3@tcp [ 948.443256] LNetError: 52730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.457629] LNet: Removed LNI 192.168.206.59@tcp [ 948.473717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.519314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.537111] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 948.541153] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 948.543035] LNet: 52738:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.660636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.679705] LNet: 52764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.685697] LNetError: 52764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.689912] LNet: Removed LNI 10.1.2.3@tcp [ 948.692307] LNetError: 52764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.707432] LNet: Removed LNI 192.168.206.59@tcp [ 948.729314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.777764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.800874] LNet: Added LNI 192.168.206.59@tcp [8/256/0/182] [ 948.808276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 948.931933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.975870] LNet: 52792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.984609] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.993616] LNet: Removed LNI 10.1.2.3@tcp [ 949.002785] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.036859] LNet: Removed LNI 192.168.206.59@tcp [ 949.064889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.145834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.175979] LNet: Added LNI 192.168.206.59@tcp [8/256/0/182] [ 949.184090] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 949.186889] LNet: 52800:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.254423] LNet: Removed LNI 10.1.2.3@tcp [ 949.263618] LNet: Removed LNI 192.168.206.59@tcp [ 949.812257] Key type .llcrypt unregistered [ 949.823618] Key type ._llcrypt unregistered [ 959.972918] Key type ._llcrypt registered [ 959.979554] Key type .llcrypt registered [ 961.441368] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:39:40 (1786732780) [ 961.817206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.841570] alg: No test for adler32 (adler32-zlib) [ 962.864132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.952181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.972063] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 963.699491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.59 [ 963.719904] LNet: 53777:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 963.733179] LNet: Removed LNI 192.168.206.59@tcp [ 969.800576] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:39:49 (1786732789) [ 970.061040] LNet: 54347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.068487] LNetError: 54347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.082982] LNet: Removed LNI 192.168.206.59@tcp [ 970.583898] Key type .llcrypt unregistered [ 970.591272] Key type ._llcrypt unregistered [ 976.375619] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 976.584529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 979.299263] Key type ._llcrypt registered [ 979.302320] Key type .llcrypt registered [ 979.505229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.526736] alg: No test for adler32 (adler32-zlib) [ 980.722332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 986.280891] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:40:05 (1786732805) [ 986.424389] LNet: 55258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.433366] LNetError: 55258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 986.475222] LNet: Removed LNI 10.1.2.3@tcp [ 986.995833] Key type .llcrypt unregistered [ 986.997955] Key type ._llcrypt unregistered [ 991.836748] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.993479] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 992.000319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 992.004408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.699481] Key type ._llcrypt registered [ 994.701781] Key type .llcrypt registered [ 994.875449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.893467] alg: No test for adler32 (adler32-zlib) [ 995.813612] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1001.257782] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:40:20 (1786732820) [ 1002.585421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1003.934264] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:40:23 (1786732823) [ 1004.145548] LNet: 56445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.151405] LNetError: 56445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1004.170348] LNet: Removed LNI 10.1.2.3@tcp [ 1004.635732] Key type .llcrypt unregistered [ 1004.637948] Key type ._llcrypt unregistered [ 1010.035795] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.252207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.259932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.269801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1012.831750] Key type ._llcrypt registered [ 1012.834189] Key type .llcrypt registered [ 1013.027318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1013.054117] alg: No test for adler32 (adler32-zlib) [ 1013.956083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.020610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1021.263958] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:40:40 (1786732840) [ 1021.754619] LNet: 57674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.764103] LNetError: 57674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1021.777970] LNet: Removed LNI 10.1.2.3@tcp [ 1022.433756] Key type .llcrypt unregistered [ 1022.436469] Key type ._llcrypt unregistered [ 1029.135928] Key type ._llcrypt registered [ 1029.140280] Key type .llcrypt registered [ 1029.526132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1029.539722] alg: No test for adler32 (adler32-zlib) [ 1030.456251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1030.480979] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1038.143081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1039.159602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_interrupt -r 1 [ 1039.233798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_interrupt -r 1 [ 1039.326190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1039.369897] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1039.685432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.713513] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1039.722135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1039.762448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.810762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1039.862552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1039.891565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.923934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.955880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.980763] LNet: 58112:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.006415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.039720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.068798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.099748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.135823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.173449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.205497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1040.816599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_dropped -r 1 [ 1040.907414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_dropped -r 1 [ 1040.968153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1041.003354] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1041.413749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.463473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1041.497412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.540924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.585631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.630030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.684805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.719391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.742447] LNet: 58221:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1041.762907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.794620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.827625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.859229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.895748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.941665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.973745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1042.557847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_aborted -r 1 [ 1042.608994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_aborted -r 1 [ 1042.650956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1042.673233] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1042.878707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.920206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1042.944743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.971963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.012325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.055462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.091909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.122913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.139468] LNet: 58330:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.157988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.190964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.224103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.246917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.288064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.346139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.391670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1044.097251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_no_route -r 1 [ 1044.139515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_no_route -r 1 [ 1044.202513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1044.534605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.575536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1044.623393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.691701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.726226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.776620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.803864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.830916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.869666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.912370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.944674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.978394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.014784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.051823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.087884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1045.766822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_timeout -r 1 [ 1045.806797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_timeout -r 1 [ 1045.835708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1045.853436] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1045.858678] LNet: Skipped 2 previous similar messages [ 1046.196956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.235893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.276635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.308707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.334275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.359926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.410374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.444972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.463975] LNet: 58549:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.469750] LNet: 58549:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1046.486401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.518952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.562322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.592325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.635331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.664852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.708686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1047.464222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_error -r 1 [ 1047.549640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_error -r 1 [ 1047.658826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1048.137605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.211350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.258696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.295883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.378966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.430567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.470541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.511037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.562912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.620743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.654772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.686410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.736814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.791266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.831388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.264533] LNet: 58699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.277752] LNetError: 58699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.304802] LNet: Removed LNI 192.168.206.59@tcp [ 1049.917702] Key type .llcrypt unregistered [ 1049.919385] Key type ._llcrypt unregistered [ 1069.396399] Key type ._llcrypt registered [ 1069.398838] Key type .llcrypt registered [ 1071.025387] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:41:29 (1786732889) [ 1071.831075] Key type .llcrypt unregistered [ 1071.832923] Key type ._llcrypt unregistered [ 1077.709154] Key type ._llcrypt registered [ 1077.710832] Key type .llcrypt registered [ 1077.846643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.856640] alg: No test for adler32 (adler32-zlib) [ 1078.789979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1078.841701] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1086.237345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1096.883414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.940679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1096.974632] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1098.121471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_interrupt -r 1 [ 1098.164320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_interrupt -r 1 [ 1098.190292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_interrupt -r 1 [ 1098.240801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_interrupt -r 1 [ 1098.285740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_interrupt -r 1 [ 1098.337118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_interrupt -r 1 [ 1098.402314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_interrupt -r 1 [ 1098.448990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_interrupt -r 1 [ 1098.501202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1098.521462] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1098.531688] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1098.721641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.747290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.770233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.800824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.828779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.859286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.887427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.912390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.932307] LNet: 60166:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.947432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.972957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.000027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.031295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.069290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.104342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.131060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1099.612778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_dropped -r 1 [ 1099.640686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_dropped -r 1 [ 1099.668712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_dropped -r 1 [ 1099.694692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_dropped -r 1 [ 1099.722985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_dropped -r 1 [ 1099.752137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_dropped -r 1 [ 1099.778812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_dropped -r 1 [ 1099.805432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_dropped -r 1 [ 1099.833804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1099.848537] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1099.861745] LNet: Skipped 2 previous similar messages [ 1100.048597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.087838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.140810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.188845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.224794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.279269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.308328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.334022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.351573] LNet: 60287:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.372044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.437679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.499590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.541775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.568619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.599682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.646124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1101.223318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_aborted -r 1 [ 1101.267440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_aborted -r 1 [ 1101.327743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_aborted -r 1 [ 1101.398466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_aborted -r 1 [ 1101.456745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_aborted -r 1 [ 1101.510774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_aborted -r 1 [ 1101.552738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_aborted -r 1 [ 1101.588396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_aborted -r 1 [ 1101.630983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1101.664670] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1101.670508] LNet: Skipped 2 previous similar messages [ 1101.929876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.964366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1101.995233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.044911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.082558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.135835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.174390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.218449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.237907] LNet: 60409:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.258827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.291251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.335887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.391549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.433498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.475267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.503584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.033533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_no_route -r 1 [ 1103.088792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_no_route -r 1 [ 1103.147625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_no_route -r 1 [ 1103.183633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_no_route -r 1 [ 1103.236267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_no_route -r 1 [ 1103.292992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_no_route -r 1 [ 1103.362607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_no_route -r 1 [ 1103.420728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_no_route -r 1 [ 1103.465778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1103.766496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.796659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.854317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.900451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.947376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.011252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.045346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.091357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.118573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.149446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.176273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.217987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.241327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.269277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.295330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.800229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_timeout -r 1 [ 1104.864781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_timeout -r 1 [ 1104.902194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_timeout -r 1 [ 1104.958490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_timeout -r 1 [ 1105.022726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_timeout -r 1 [ 1105.061715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_timeout -r 1 [ 1105.097196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_timeout -r 1 [ 1105.123960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_timeout -r 1 [ 1105.183939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1105.212068] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1105.219941] LNet: Skipped 5 previous similar messages [ 1105.427941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.468551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.515539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.538063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.565969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.595474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.632319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.673486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.693930] LNet: 60652:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.699118] LNet: 60652:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1105.726247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.759321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.792122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.824735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.852231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.876448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.906482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.476849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_error -r 1 [ 1106.519482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_error -r 1 [ 1106.561987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_error -r 1 [ 1106.608747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_error -r 1 [ 1106.673805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_error -r 1 [ 1106.713089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_error -r 1 [ 1106.774390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_error -r 1 [ 1106.818499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_error -r 1 [ 1106.857706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1107.125799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.157798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.181884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.204126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.231569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.263627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.293643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.336104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.377488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.409578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.467582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.505583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.552503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.599272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.649168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1113.373947] LNet: 60849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.379790] LNetError: 60849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.387686] LNet: Removed LNI 192.168.206.59@tcp [ 1114.414200] LNet: Removed LNI 192.168.206.59@tcp1 [ 1114.725729] Key type .llcrypt unregistered [ 1114.732133] Key type ._llcrypt unregistered [ 1130.724535] Key type ._llcrypt registered [ 1130.727065] Key type .llcrypt registered [ 1131.847905] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:42:30 (1786732950) [ 1132.867258] Key type .llcrypt unregistered [ 1132.871195] Key type ._llcrypt unregistered [ 1139.259882] Key type ._llcrypt registered [ 1139.262343] Key type .llcrypt registered [ 1139.492775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.519258] alg: No test for adler32 (adler32-zlib) [ 1140.510917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1140.551383] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1148.180536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1149.055657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_dropped -r 1 [ 1149.153671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_dropped -r 1 [ 1149.233181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1149.274556] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1149.547887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.590204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.625297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.648827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.673352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.701444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.751861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.799891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.819989] LNet: 62100:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.835544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.891897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.919639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.947425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.981483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.010524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.044482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1150.579424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_error -r 1 [ 1150.624193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_error -r 1 [ 1150.655413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1150.673208] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1150.887814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.925776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.965572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.993311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.017551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.041840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.068487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.104455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.121248] LNet: 62209:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.139809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.167624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.199119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.224495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.247614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.269530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.299928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1151.772037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_timeout -r 1 [ 1151.837290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_timeout -r 1 [ 1151.889659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1151.914977] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1152.131703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.183476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.218989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.258716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.300640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.326518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.362440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.404392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.428601] LNet: 62318:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.461889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.495909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.545253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.577950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.617369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.655503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.688857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1152.917156] LNet: 62359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.926247] LNetError: 62359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.945130] LNet: Removed LNI 192.168.206.59@tcp [ 1153.412777] Key type .llcrypt unregistered [ 1153.417256] Key type ._llcrypt unregistered [ 1168.120590] Key type ._llcrypt registered [ 1168.123902] Key type .llcrypt registered [ 1169.387446] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:43:08 (1786732988) [ 1170.147950] Key type .llcrypt unregistered [ 1170.154598] Key type ._llcrypt unregistered [ 1176.426356] Key type ._llcrypt registered [ 1176.427953] Key type .llcrypt registered [ 1176.623398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1176.654732] alg: No test for adler32 (adler32-zlib) [ 1177.577536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1177.603696] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1184.755187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1193.499626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1193.540517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1193.566894] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1194.788405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_dropped -r 1 [ 1194.821541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e remote_dropped -r 1 [ 1194.857030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_dropped -r 1 [ 1194.891579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e remote_dropped -r 1 [ 1194.931210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e remote_dropped -r 1 [ 1194.987403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e remote_dropped -r 1 [ 1195.049144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e remote_dropped -r 1 [ 1195.120385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e remote_dropped -r 1 [ 1195.178636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1195.198057] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1195.207082] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.159@tcp [ 1195.456144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1195.505491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.551275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.589482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.631241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.667949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.699537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.735784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.759678] LNet: 63827:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.783597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.823824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.868890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.910939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.952739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.986660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.020930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.052628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.555297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_error -r 1 [ 1196.588706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e remote_error -r 1 [ 1196.632230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_error -r 1 [ 1196.672989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e remote_error -r 1 [ 1196.705490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e remote_error -r 1 [ 1196.732241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e remote_error -r 1 [ 1196.758939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e remote_error -r 1 [ 1196.795902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e remote_error -r 1 [ 1196.827233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1196.851602] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1196.855401] LNet: Skipped 2 previous similar messages [ 1197.048764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.096895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.135189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.174062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.204549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.229377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.259907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.291904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.308449] LNet: 63950:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.323135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.348872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.378757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.402907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.426760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.456132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.498642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.536248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.997676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_timeout -r 1 [ 1198.025906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e remote_timeout -r 1 [ 1198.084884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_timeout -r 1 [ 1198.122959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e remote_timeout -r 1 [ 1198.160623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e remote_timeout -r 1 [ 1198.183918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e remote_timeout -r 1 [ 1198.212540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e remote_timeout -r 1 [ 1198.269929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e remote_timeout -r 1 [ 1198.312180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.159@tcp [ 1198.334986] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1198.341054] LNet: Skipped 1 previous similar message [ 1198.499423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.534809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1198.574240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.629525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.682764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.728961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.759919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.790847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.809184] LNet: 64073:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.821547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.844475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.865409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.887563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.910657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.936856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.960142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.986345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.220513] LNet: 64151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.226343] LNetError: 64151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.233179] LNet: Removed LNI 192.168.206.59@tcp [ 1203.244847] LNet: Removed LNI 192.168.206.59@tcp1 [ 1203.683563] Key type .llcrypt unregistered [ 1203.688587] Key type ._llcrypt unregistered [ 1219.651676] Key type ._llcrypt registered [ 1219.654504] Key type .llcrypt registered [ 1220.977784] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:43:59 (1786733039) [ 1222.556991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1224.129326] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:44:03 (1786733043) [ 1225.093704] Key type .llcrypt unregistered [ 1225.096438] Key type ._llcrypt unregistered [ 1231.848776] Key type ._llcrypt registered [ 1231.852317] Key type .llcrypt registered [ 1232.131702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1232.164807] alg: No test for adler32 (adler32-zlib) [ 1233.351723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1233.371960] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1242.032721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1242.796378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e network_timeout -r 1 [ 1242.822714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e network_timeout -r 1 [ 1242.872303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1242.891787] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1243.130642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.155993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.179441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.218865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.250291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.280036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.313787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.354559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.382657] LNet: 65696:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.399328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.441261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.501762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.532403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.570746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.616549] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1243.620707] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1243.647077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.948535] LNet: 65735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.953777] LNetError: 65735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.975924] LNet: Removed LNI 192.168.206.59@tcp [ 1244.462513] Key type .llcrypt unregistered [ 1244.464735] Key type ._llcrypt unregistered [ 1262.583460] Key type ._llcrypt registered [ 1262.588630] Key type .llcrypt registered [ 1262.964654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1262.981908] alg: No test for adler32 (adler32-zlib) [ 1263.906983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1263.932794] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1271.510928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1279.702638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.757169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1279.791988] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1281.139591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e network_timeout -r 1 [ 1281.186272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e network_timeout -r 1 [ 1281.221601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e network_timeout -r 1 [ 1281.261818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e network_timeout -r 1 [ 1281.294505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e network_timeout -r 1 [ 1281.347726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e network_timeout -r 1 [ 1281.382385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e network_timeout -r 1 [ 1281.414282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e network_timeout -r 1 [ 1281.466170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1281.505269] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1281.694273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.731916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.776756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.806644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.875688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.926699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.953336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.987844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.005658] LNet: 66615:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.025481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.092529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.146136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.215989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.256211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.296763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.493378] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1287.525037] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1287.856811] LNet: 66690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.863519] LNetError: 66690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.880352] LNet: Removed LNI 192.168.206.59@tcp [ 1287.906234] LNet: Removed LNI 192.168.206.59@tcp1 [ 1288.281816] Key type .llcrypt unregistered [ 1288.284685] Key type ._llcrypt unregistered [ 1305.894192] Key type ._llcrypt registered [ 1305.898205] Key type .llcrypt registered [ 1307.382418] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:45:26 (1786733126) [ 1307.886518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.912505] alg: No test for adler32 (adler32-zlib) [ 1308.796526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.843746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.862642] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1309.497193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1309.518992] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1310.283144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 1310.327860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1310.359087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1310.399485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1310.438692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1310.482184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1310.525972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.59@tcp [ 1310.546595] LNetError: 67792:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.59@tcp to recovery [ 1310.881134] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1310.883183] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1312.928615] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1314.091072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1317.025598] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1317.615621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1317.661724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1317.731289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1317.746226] LNet: 67899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.752587] LNetError: 67899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.816246] LNet: Removed LNI 192.168.206.59@tcp [ 1318.824459] LNetError: 67899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.837908] LNet: Removed LNI 192.168.206.59@tcp1 [ 1318.886136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.966141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1319.003951] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1319.764684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1319.786206] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1320.635427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 1320.661470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.714295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1320.764771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1320.802108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1320.837995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1320.881699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1320.912737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.59@tcp [ 1320.933907] LNetError: 68172:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.59@tcp to recovery [ 1320.992488] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1322.225360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1326.112943] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1326.120240] LNet: Skipped 1 previous similar message [ 1331.289179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1331.331389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1331.397489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.244447] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:45:56 (1786733156) [ 1337.366234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1337.402993] LNet: 68875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.414852] LNetError: 68875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.464525] LNet: Removed LNI 192.168.206.59@tcp [ 1338.475730] LNet: Removed LNI 192.168.206.59@tcp1 [ 1338.534060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.623680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1338.663342] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1339.229298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.261642] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1339.950555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 1340.005485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1340.030327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1340.054993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1340.087974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1340.119878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1340.151582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.59@tcp --health 0 [ 1340.228307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1340.643670] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.59@tcp [ 1341.664359] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1341.672535] LNet: Skipped 1 previous similar message [ 1356.756466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1356.914780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1356.963810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.062349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.59@tcp --health 500 [ 1357.240830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.979265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1364.101507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.122433] LNet: 69383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.128890] LNetError: 69383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.133400] LNetError: 69383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1364.137949] LNet: Removed LNI 192.168.206.59@tcp [ 1364.151253] LNet: Removed LNI 192.168.206.59@tcp1 [ 1364.167302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.205337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1364.227707] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1365.275169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1365.302962] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1366.417463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 1366.497875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.587674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1366.641488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1366.730316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1366.811455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1366.880522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.924801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.59@tcp --health 0 [ 1367.265513] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1367.279339] LNet: Skipped 3 previous similar messages [ 1368.358790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.606595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1378.629239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1378.648987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1386.612804] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:46:45 (1786733205) [ 1387.021683] LNet: 70429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.026876] LNetError: 70429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.033732] LNetError: 70429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1387.038653] LNet: Removed LNI 192.168.206.59@tcp [ 1387.053175] LNet: Removed LNI 192.168.206.59@tcp1 [ 1387.565851] Key type .llcrypt unregistered [ 1387.571584] Key type ._llcrypt unregistered [ 1395.307556] Key type ._llcrypt registered [ 1395.308927] Key type .llcrypt registered [ 1395.479627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1395.502148] alg: No test for adler32 (adler32-zlib) [ 1396.433442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1396.468321] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1406.532980] Lustre: DEBUG MARKER: Initial discovery [ 1406.566634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.159@tcp [ 1408.923659] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1409.083843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1409.161469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.159@tcp [ 1409.244659] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1409.298812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1409.331805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.159@tcp -r 1 -m GET -l 3 [ 1409.364250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.159@tcp -r 1 -m GET -e local_error [ 1409.398456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.159@tcp -r 1 -m PUT -l 6 [ 1409.831699] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1410.660206] Lustre: DEBUG MARKER: Force oleg659-client.virtnet to discover 192.168.206.159@tcp (in background) [ 1410.835426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.159@tcp [ 1412.152423] Lustre: DEBUG MARKER: Force oleg659-server to discover 192.168.206.59@tcp [ 1413.920375] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1414.592994] Lustre: DEBUG MARKER: Wait for 71189 [ 1415.984413] Lustre: DEBUG MARKER: Finished wait on 71189 [ 1416.095538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1416.130685] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1416.148655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1416.384539] LNet: 71642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.392984] LNetError: 71642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.456359] LNet: Removed LNI 192.168.206.59@tcp [ 1417.949806] Key type .llcrypt unregistered [ 1417.952403] Key type ._llcrypt unregistered [ 1431.176926] Key type ._llcrypt registered [ 1431.178819] Key type .llcrypt registered [ 1432.150835] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:47:31 (1786733251) [ 1432.500282] Key type .llcrypt unregistered [ 1432.501986] Key type ._llcrypt unregistered [ 1437.648548] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1437.785739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1437.789824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1437.796801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1440.276288] Key type ._llcrypt registered [ 1440.279209] Key type .llcrypt registered [ 1440.448744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1440.461394] alg: No test for adler32 (adler32-zlib) [ 1441.365123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.433957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1441.477595] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1442.092894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1442.109840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1448.900128] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:47:48 (1786733268) [ 1449.050832] LNet: 73707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.057098] LNetError: 73707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1449.062280] LNet: Removed LNI 10.1.2.3@tcp [ 1449.078189] LNet: Removed LNI 192.168.206.59@tcp [ 1449.869465] Key type .llcrypt unregistered [ 1449.872687] Key type ._llcrypt unregistered [ 1456.055633] Key type ._llcrypt registered [ 1456.060523] Key type .llcrypt registered [ 1456.262249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1456.275193] alg: No test for adler32 (adler32-zlib) [ 1457.280901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.373258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.420819] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1458.340664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1458.364414] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1459.326168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.59@tcp [ 1460.384641] LNetError: Unexpected error -22 connecting to 192.168.206.59@tcp at host 192.168.206.59:988 [ 1465.429346] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:48:04 (1786733284) [ 1465.522382] LNet: 74836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.531110] LNetError: 74836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1465.541379] LNet: Removed LNI 10.1.2.3@tcp [ 1465.558731] LNet: Removed LNI 192.168.206.59@tcp [ 1466.281788] Key type .llcrypt unregistered [ 1466.285852] Key type ._llcrypt unregistered [ 1471.736348] Key type ._llcrypt registered [ 1471.739900] Key type .llcrypt registered [ 1471.891955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1471.904131] alg: No test for adler32 (adler32-zlib) [ 1472.871562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.984200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1473.017636] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1473.824441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1473.850900] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 1474.769187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.59@tcp1 --nid 192.168.206.59@tcp2 [ 1478.991069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp1 192.168.206.59@tcp2 [ 1479.009527] LNetError: 75681:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.59@tcp1 to 192.168.206.59@tcp2 - Route table may be misconfigured [ 1479.029936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp2 192.168.206.59@tcp1 [ 1484.990065] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:48:24 (1786733304) [ 1485.378266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.415831] LNet: 76254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.420910] LNetError: 76254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.426976] LNet: Removed LNI 192.168.206.59@tcp1 [ 1486.453033] LNet: Removed LNI 192.168.206.59@tcp2 [ 1486.529832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.608857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.667388] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1487.399749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1487.415036] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1488.534967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 1488.663924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.59@tcp [ 1488.756529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp1 192.168.206.59@tcp1 [ 1488.853684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.59@tcp -d 192.168.206.59@tcp -e network_timeout [ 1488.987876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -e network_timeout [ 1489.083453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -e network_timeout [ 1489.165780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -e network_timeout [ 1489.236562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1489.267336] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1489.295850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1489.480950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1489.632732] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1489.650345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1498.463812] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:48:36 (1786733316) [ 1498.623486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.657679] LNet: 77111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.672174] LNetError: 77111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.677844] LNetError: 77111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1498.689934] LNet: Removed LNI 192.168.206.59@tcp [ 1499.753413] LNet: Removed LNI 192.168.206.59@tcp1 [ 1499.805872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.881297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1500.189217] LNet: 77154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.666785] Key type .llcrypt unregistered [ 1500.668801] Key type ._llcrypt unregistered [ 1512.132243] Key type ._llcrypt registered [ 1512.135254] Key type .llcrypt registered [ 1513.340098] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:48:52 (1786733332) [ 1513.738725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.767294] alg: No test for adler32 (adler32-zlib) [ 1514.734858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.969753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1517.710955] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:48:56 (1786733336) [ 1517.865293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.912602] LNet: 78249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.956149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.047621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.068367] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1518.919355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.946468] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1519.677369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1519.704365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp1 [ 1519.758706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp1 [ 1525.972789] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:49:04 (1786733344) [ 1527.653125] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg659-server [ 1529.354949] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:49:08 (1786733348) [ 1531.066231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg659-server [ 1533.048472] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:49:11 (1786733351) [ 1534.935231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg659-server [ 1536.636717] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:49:15 (1786733355) [ 1538.737649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg659-server [ 1540.369412] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:49:19 (1786733359) [ 1542.295618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg659-server [ 1544.152052] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:49:22 (1786733362) [ 1546.339409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg659-server [ 1547.995469] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:49:26 (1786733366) [ 1549.724388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg659-server [ 1551.569199] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:49:30 (1786733370) [ 1553.013112] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg659-server [ 1555.099790] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:49:34 (1786733374) [ 1556.948202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg659-server [ 1558.736429] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:49:37 (1786733377) [ 1558.878918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.914730] LNet: 82129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.924447] LNetError: 82129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.937280] LNet: Removed LNI 192.168.206.59@tcp [ 1559.975080] LNet: Removed LNI 192.168.206.59@tcp1 [ 1560.028927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.119601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.147395] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1560.879582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1560.937747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1561.080366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.095837] LNet: 82279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.104424] LNetError: 82279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.108665] LNetError: 82279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1561.124758] LNet: Removed LNI 192.168.206.59@tcp [ 1561.138865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.190758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1561.228549] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1561.921335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1562.107210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.120383] LNetError: 82424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.140303] LNet: Removed LNI 192.168.206.59@tcp [ 1562.156185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.206856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.257566] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1562.894726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1562.967694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1563.109406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.141244] LNet: 82571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.145802] LNet: 82571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1564.204865] LNet: Removed LNI 192.168.206.59@tcp [ 1564.240976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.331621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1564.370601] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1565.016607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1565.248362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.274576] LNetError: 82716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.278906] LNetError: 82716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1565.296940] LNet: Removed LNI 192.168.206.59@tcp [ 1565.323919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.428840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.470910] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1566.422359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1566.535414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1566.791150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.854672] LNet: Removed LNI 192.168.206.59@tcp [ 1567.909655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.040280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1568.083805] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1569.118625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1569.400736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.444076] LNet: 83009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.452702] LNet: 83009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1569.461330] LNetError: 83009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.470839] LNetError: 83009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.487069] LNet: Removed LNI 192.168.206.59@tcp [ 1569.550743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.649913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.696837] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1570.523694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1570.621719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1570.873532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.942939] LNet: Removed LNI 192.168.206.59@tcp [ 1571.963965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.035845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1572.057092] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1572.849471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1573.120681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.186969] LNet: Removed LNI 192.168.206.59@tcp [ 1574.210963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.276209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.304102] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1574.859731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1574.932341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1575.205687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.295224] LNet: Removed LNI 192.168.206.59@tcp [ 1576.323309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.411464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1576.438438] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1577.176670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1577.406732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.479647] LNet: Removed LNI 192.168.206.59@tcp [ 1578.521987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.599050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.657375] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1579.483837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1579.575574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1579.992240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.029660] LNet: 83742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.034191] LNet: 83742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1580.045992] LNetError: 83742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.055389] LNetError: 83742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1580.103320] LNet: Removed LNI 192.168.206.59@tcp [ 1580.141698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.238610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1580.297189] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1581.245703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1581.560843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.648696] LNet: Removed LNI 192.168.206.59@tcp [ 1582.687963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.749700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.767088] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1583.526750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1583.587267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.629573] LNet: Removed LNI 192.168.206.59@tcp [ 1583.652949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.726662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.750391] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1584.624676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1584.737828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.777855] LNet: Removed LNI 192.168.206.59@tcp [ 1584.811526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.937926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1585.070968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.119422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.185427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1585.290138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.327108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.413335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1585.526976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.561599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1585.650749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1585.708200] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1586.558280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.588030] LNet: Removed LNI 192.168.206.59@tcp [ 1586.604597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.656685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1592.539437] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:50:11 (1786733411) [ 1592.860068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.935650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.005348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1593.027684] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/180] [ 1594.253577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.309496] LNet: Removed LNI 192.168.206.59@tcp231 [ 1594.351506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.476324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1594.515556] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/180] [ 1594.524992] LNet: 85055:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1595.042944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1595.092200] LNet: Removed LNI 192.168.206.59@tcp231 [ 1595.108754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1595.132805] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/180] [ 1596.309305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.337621] LNet: 85199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.342614] LNet: 85199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1596.347651] LNetError: 85199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.353672] LNetError: 85199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1596.369794] LNet: Removed LNI 192.168.206.59@tcp231 [ 1596.386415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.459459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1596.480724] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/0] [ 1596.485876] LNet: 85207:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1596.721249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1596.751686] LNet: Removed LNI 192.168.206.59@tcp231 [ 1596.762423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1596.779748] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/0] [ 1598.043272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.064324] LNet: Removed LNI 192.168.206.59@tcp231 [ 1598.085672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.149545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1598.192788] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/60] [ 1598.203768] LNet: 85358:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1598.830058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1598.885291] LNet: Removed LNI 192.168.206.59@tcp231 [ 1598.916834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1598.966870] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/60] [ 1600.332635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.373283] LNet: Removed LNI 192.168.206.59@tcp231 [ 1600.405771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.484436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1600.501542] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/180] [ 1600.505157] LNet: 85510:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1600.972876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1601.014484] LNet: Removed LNI 192.168.206.59@tcp231 [ 1601.041802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1601.061372] LNet: Added LNI 192.168.206.59@tcp231 [8/256/0/180] [ 1610.037635] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:50:28 (1786733428) [ 1610.116366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.169290] LNet: Removed LNI 192.168.206.59@tcp231 [ 1610.195368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.283253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1610.319408] LNet: Added LNI 192.168.206.59@tcp232 [8/256/0/180] [ 1611.319295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1611.377157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1611.447784] LNet: Removed LNI 192.168.206.59@tcp232 [ 1611.465050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1611.486889] LNet: Added LNI 192.168.206.59@tcp232 [8/256/0/180] [ 1612.449691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1612.492149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1612.529322] LNet: Removed LNI 192.168.206.59@tcp232 [ 1612.554406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1612.587308] LNet: Added LNI 192.168.206.59@tcp232 [8/256/0/180] [ 1613.523456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1613.583971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1613.629896] LNet: Removed LNI 192.168.206.59@tcp232 [ 1613.664830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1613.699038] LNet: Added LNI 192.168.206.59@tcp232 [8/256/0/180] [ 1614.632560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1614.680641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1614.743813] LNet: Removed LNI 192.168.206.59@tcp232 [ 1614.805117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1622.504370] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:50:41 (1786733441) [ 1623.575293] Key type .llcrypt unregistered [ 1623.580509] Key type ._llcrypt unregistered [ 1629.325456] Key type ._llcrypt registered [ 1629.327722] Key type .llcrypt registered [ 1629.543725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1629.559803] alg: No test for adler32 (adler32-zlib) [ 1630.495325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1630.517299] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1637.330631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1644.831979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.901588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1644.939974] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1646.409654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_interrupt -r 2 [ 1646.451446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_interrupt -r 2 [ 1646.518255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_interrupt -r 2 [ 1646.585463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_interrupt -r 2 [ 1646.654440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_interrupt -r 2 [ 1646.682485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_interrupt -r 2 [ 1646.732359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_interrupt -r 2 [ 1646.770173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_interrupt -r 2 [ 1646.799608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.159@tcp [ 1646.827252] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1646.838350] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1647.078956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1647.108363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1647.132343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1647.175477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1647.219551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1647.252336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1647.279552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1647.329472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1647.344480] LNet: 88012:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1647.376456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1647.428404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1647.494128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1647.560069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1647.590739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1647.637320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1648.112036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_dropped -r 2 [ 1648.153147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_dropped -r 2 [ 1648.196701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_dropped -r 2 [ 1648.242662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_dropped -r 2 [ 1648.295528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_dropped -r 2 [ 1648.338343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_dropped -r 2 [ 1648.376238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_dropped -r 2 [ 1648.429861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_dropped -r 2 [ 1648.471946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.159@tcp [ 1648.490210] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1648.726449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1648.758242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1648.798935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.838160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.866363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.912208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1648.941636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.979875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1648.998304] LNet: 88132:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1649.014225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1649.040450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1649.073577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1649.100500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1649.135916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1649.157029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.572672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_aborted -r 2 [ 1649.601084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_aborted -r 2 [ 1649.638716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_aborted -r 2 [ 1649.672529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_aborted -r 2 [ 1649.701344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_aborted -r 2 [ 1649.750310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_aborted -r 2 [ 1649.820768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_aborted -r 2 [ 1649.898977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_aborted -r 2 [ 1649.933480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.159@tcp [ 1649.951509] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1650.172805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1650.197697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1650.243897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1650.285673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.314305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.343584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.383296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.414577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.434568] LNet: 88252:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.451869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.494377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.527879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.554915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.584902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.627832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1651.137800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_no_route -r 2 [ 1651.170353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_no_route -r 2 [ 1651.214235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_no_route -r 2 [ 1651.243989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_no_route -r 2 [ 1651.279066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_no_route -r 2 [ 1651.307837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_no_route -r 2 [ 1651.346363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_no_route -r 2 [ 1651.385286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_no_route -r 2 [ 1651.422551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.159@tcp [ 1651.666439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1651.698064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1651.719769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1651.756420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1651.778862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1651.804631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1651.837103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1651.863393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1651.886282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1651.923073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1651.954325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1651.994358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.032523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.057968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1652.498130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e local_timeout -r 2 [ 1652.542830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e local_timeout -r 2 [ 1652.589173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e local_timeout -r 2 [ 1652.658892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e local_timeout -r 2 [ 1652.718719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e local_timeout -r 2 [ 1652.762369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e local_timeout -r 2 [ 1652.822513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e local_timeout -r 2 [ 1652.851554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e local_timeout -r 2 [ 1652.891112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.159@tcp [ 1652.910795] LNet: There was an unexpected network error while writing to 192.168.206.159: rc = -22 [ 1652.914782] LNet: Skipped 1 previous similar message [ 1653.129925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1653.161277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1653.190779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1653.217576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1653.245195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1653.283512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1653.316535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1653.346983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1653.365565] LNet: 88493:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.370599] LNet: 88493:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1653.387262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1653.423807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1653.452829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1653.476425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1653.500935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1653.530646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1653.965623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp -m GET -e remote_dropped -r 2 [ 1654.009416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.159@tcp1 -m GET -e remote_dropped -r 2 [ 1654.050954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp -m GET -e remote_dropped -r 2 [ 1654.115630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp -d 192.168.206.59@tcp1 -m GET -e remote_dropped -r 2 [ 1654.162962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp -m GET -e remote_dropped -r 2 [ 1654.215326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.159@tcp1 -m GET -e remote_dropped -r 2 [ 1654.265496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp -m GET -e remote_dropped -r 2 [ 1654.322325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.59@tcp1 -d 192.168.206.59@tcp1 -m GET -e remote_dropped -r 2 [ 1654.354787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.59@tcp 192.168.206.159@tcp [ 1654.379264] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.159@tcp [ 1654.595651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1654.625957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1654.660326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1654.695409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1654.729210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1654.761666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1654.789439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1654.822597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1654.853679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1654.894610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1654.933652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1654.963655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1654.994651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1655.030569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1659.735314] LNet: 88687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.740753] LNetError: 88687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.752521] LNet: Removed LNI 192.168.206.59@tcp [ 1659.768641] LNet: Removed LNI 192.168.206.59@tcp1 [ 1660.169246] Key type .llcrypt unregistered [ 1660.179949] Key type ._llcrypt unregistered [ 1675.359513] Key type ._llcrypt registered [ 1675.361838] Key type .llcrypt registered [ 1676.677189] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:51:35 (1786733495) [ 1677.235508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.250809] alg: No test for adler32 (adler32-zlib) [ 1678.171472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.212482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.240483] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1678.947128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1678.968641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.59@tcp [ 1678.990072] LNetError: 89667:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.59@tcp to recovery [ 1679.066163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.59@tcp [ 1679.137303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.588989] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:51:44 (1786733504) [ 1686.775550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.801677] LNet: 90251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.812740] LNetError: 90251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.825891] LNet: Removed LNI 192.168.206.59@tcp [ 1686.873047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.967059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.010316] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1687.744074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1687.766857] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 1688.761504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.59@tcp [ 1688.779608] LNetError: 90525:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.59@tcp to recovery [ 1688.993720] LNet: 1 local NIs in recovery (showing 1): 192.168.206.59@tcp [ 1689.013477] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1690.021693] LNet: There was an unexpected network error while writing to 192.168.206.59: rc = -22 [ 1690.044426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.59@tcp [ 1702.476244] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:52:01 (1786733521) [ 1704.339686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg659-server [ 1705.959547] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:52:04 (1786733524) [ 1707.990770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg659-server [ 1709.943785] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:52:08 (1786733528) [ 1711.850411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg659-server [ 1713.601350] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:52:12 (1786733532) [ 1713.996425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.022607] LNet: 92172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.027960] LNetError: 92172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.040380] LNet: Removed LNI 192.168.206.59@tcp [ 1715.045878] LNetError: 92172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.065921] LNet: Removed LNI 192.168.206.59@tcp1 [ 1715.105185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.184750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.227192] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1715.929568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1716.003985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1716.061763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1716.133980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1716.180161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1716.249495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1723.553371] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:52:22 (1786733542) [ 1723.747190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.785631] LNet: 92897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.800976] LNetError: 92897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.826721] LNet: Removed LNI 192.168.206.59@tcp [ 1723.849920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.964438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.021198] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1731.832437] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:52:30 (1786733550) [ 1731.980394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.001666] LNet: 93599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.013538] LNetError: 93599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.031101] LNet: Removed LNI 192.168.206.59@tcp [ 1732.047774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.090450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.110799] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1739.300536] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:52:38 (1786733558) [ 1740.771076] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1742.842211] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:52:41 (1786733561) [ 1743.372684] LNet: 94637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.377904] LNetError: 94637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.394274] LNet: Removed LNI 192.168.206.59@tcp [ 1744.219692] Key type .llcrypt unregistered [ 1744.225801] Key type ._llcrypt unregistered [ 1750.776375] Key type ._llcrypt registered [ 1750.779863] Key type .llcrypt registered [ 1751.032732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1751.068379] alg: No test for adler32 (adler32-zlib) [ 1752.018431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1752.058621] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1759.523791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 1764.211990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.159@tcp [ 1764.437615] LNet: 95055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.446858] LNetError: 95055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.462922] LNet: Removed LNI 192.168.206.59@tcp [ 1765.075451] Key type .llcrypt unregistered [ 1765.077172] Key type ._llcrypt unregistered [ 1778.092762] Key type ._llcrypt registered [ 1778.095702] Key type .llcrypt registered [ 1779.827807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1781.943147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1783.797629] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:53:22 (1786733602) [ 1785.162940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1786.971470] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:53:25 (1786733605) [ 1789.013495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg659-server [ 1790.784401] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:53:29 (1786733609) [ 1792.837404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg659-server [ 1794.479284] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:53:33 (1786733613) [ 1796.272167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1796.323116] alg: No test for adler32 (adler32-zlib) [ 1797.312364] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 1797.485856] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1799.144178] Key type lgssc registered [ 1800.864147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1822.799504] Key type lgssc unregistered [ 1823.078108] LNet: 98231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.093303] LNetError: 98231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.107187] LNet: Removed LNI 192.168.206.59@tcp [ 1823.784355] Key type .llcrypt unregistered [ 1823.790268] Key type ._llcrypt unregistered [ 1839.379446] Key type ._llcrypt registered [ 1839.381815] Key type .llcrypt registered [ 1840.745376] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:54:19 (1786733659) [ 1841.223839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.245902] alg: No test for adler32 (adler32-zlib) [ 1842.227920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.319646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1842.377884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1842.451270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1842.536208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1849.351411] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:54:28 (1786733668) [ 1849.547121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.576108] LNet: 99644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.609699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.691174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1849.709027] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1850.625081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.647196] LNet: 99791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.654117] LNetError: 99791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.673754] LNet: Removed LNI 192.168.206.59@tcp [ 1850.711389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.777106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1850.812380] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1851.376826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.411690] LNetError: 99820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.452866] LNet: Removed LNI 192.168.206.59@tcp [ 1851.472568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.522466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1851.557546] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1858.412930] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:54:37 (1786733677) [ 1858.823657] LNet: 100410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.840604] LNet: 100410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1858.850125] LNetError: 100410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.865858] LNet: Removed LNI 192.168.206.59@tcp [ 1859.661502] Key type .llcrypt unregistered [ 1859.665246] Key type ._llcrypt unregistered [ 1867.120948] Key type ._llcrypt registered [ 1867.124769] Key type .llcrypt registered [ 1867.444755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1867.469834] alg: No test for adler32 (adler32-zlib) [ 1868.458147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.604095] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1868.610619] LNet: *** cfs_fail_loc=e003, val=100*** [ 1869.571912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1869.591544] LNet: *** cfs_fail_loc=e003, val=100*** [ 1869.595631] LNet: Skipped 2 previous similar messages [ 1869.603274] Ping reply received with mapping: [ 1869.605926] found metadata reply NI[192.168.206.59@tcp] [ 1869.609337] 3b ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1869.612243] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1869.614867] 5a 5a 5a 5a 5a 5a 5a 5a [ 1869.792507] LNet: 100808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.809425] LNetError: 100808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.832736] LNet: Removed LNI 192.168.206.59@tcp [ 1870.662575] Key type .llcrypt unregistered [ 1870.664457] Key type ._llcrypt unregistered [ 1884.052168] Key type ._llcrypt registered [ 1884.056988] Key type .llcrypt registered [ 1885.493053] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:55:04 (1786733704) [ 1886.195260] Key type .llcrypt unregistered [ 1886.199741] Key type ._llcrypt unregistered [ 1892.607137] Key type ._llcrypt registered [ 1892.609908] Key type .llcrypt registered [ 1892.841454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1892.865826] alg: No test for adler32 (adler32-zlib) [ 1893.996496] LNetError: 101855:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1894.382761] Key type .llcrypt unregistered [ 1894.387341] Key type ._llcrypt unregistered [ 1899.946904] Key type ._llcrypt registered [ 1899.950251] Key type .llcrypt registered [ 1901.185702] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:55:20 (1786733720) [ 1901.799375] Key type .llcrypt unregistered [ 1901.802490] Key type ._llcrypt unregistered [ 1907.762959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1907.768160] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1910.397660] Key type ._llcrypt registered [ 1910.399852] Key type .llcrypt registered [ 1910.542111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1910.553124] alg: No test for adler32 (adler32-zlib) [ 1911.442612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.538126] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1920.557549] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:55:39 (1786733739) [ 1920.654786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.682371] LNet: 103629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.689153] LNetError: 103629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1920.712335] LNet: Removed LNI 10.1.2.3@tcp [ 1920.761960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.488656] Key type .llcrypt unregistered [ 1921.492848] Key type ._llcrypt unregistered [ 1928.684713] Key type ._llcrypt registered [ 1928.692479] Key type .llcrypt registered [ 1928.882084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1928.904576] alg: No test for adler32 (adler32-zlib) [ 1929.839554] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1929.923439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1930.149986] LNet: 103926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.155951] LNetError: 103926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.179109] LNet: Removed LNI 192.168.206.59@tcp [ 1931.966487] Key type .llcrypt unregistered [ 1931.970441] Key type ._llcrypt unregistered [ 1943.660274] Key type ._llcrypt registered [ 1943.667205] Key type .llcrypt registered [ 1944.902400] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:56:03 (1786733763) [ 1945.727321] Key type .llcrypt unregistered [ 1945.729850] Key type ._llcrypt unregistered [ 1952.022981] Key type ._llcrypt registered [ 1952.025484] Key type .llcrypt registered [ 1952.208486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1952.221851] alg: No test for adler32 (adler32-zlib) [ 1959.752756] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:56:18 (1786733778) [ 1959.917177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.987891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.125579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.174768] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1961.153770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1961.247991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1961.283449] LNet: Removed LNI 192.168.206.59@tcp [ 1961.559482] LNet: 105670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.137460] Key type .llcrypt unregistered [ 1962.140713] Key type ._llcrypt unregistered [ 1974.556463] Key type ._llcrypt registered [ 1974.562936] Key type .llcrypt registered [ 1975.965169] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:56:34 (1786733794) [ 1976.307674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.322379] alg: No test for adler32 (adler32-zlib) [ 1977.291690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.363684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.387654] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 1977.950269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 1977.976372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.59@tcp [ 1977.991782] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1977.996849] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.001031] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1977, 0, 0 [ 1978.009765] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1978.014259] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1978.019318] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.022906] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1977, 0, 0 [ 1978.028393] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1978.031443] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1978.035655] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.040248] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1977, 0, 2027 [ 1978.044558] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1978.047361] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1978.053992] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1978.057463] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1977, 0, 2027 [ 1978.061662] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1984.042411] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:56:42 (1786733802) [ 1984.541928] LNet: 107211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.557849] LNetError: 107211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.576071] LNet: Removed LNI 192.168.206.59@tcp [ 1985.190454] Key type .llcrypt unregistered [ 1985.199052] Key type ._llcrypt unregistered [ 1992.838152] Key type ._llcrypt registered [ 1992.842887] Key type .llcrypt registered [ 1993.122763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1993.149675] alg: No test for adler32 (adler32-zlib) [ 1994.050535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1994.073072] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2002.180602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 2011.648908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.747493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2011.798200] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 2013.170576] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2013.317443] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2013.320773] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2013.330510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2015.939167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2015.984455] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2027.196573] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:57:25 (1786733845) [ 2027.333462] LNet: 108711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.342994] LNetError: 108711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.351468] LNet: Removed LNI 192.168.206.59@tcp [ 2028.384287] LNet: Removed LNI 192.168.206.59@tcp1 [ 2028.389748] LNetError: 108711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2028.400205] LNetError: 108711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2028.424400] LNet: Removed LNI 10.1.2.3@tcp99 [ 2029.148512] Key type .llcrypt unregistered [ 2029.151099] Key type ._llcrypt unregistered [ 2036.230818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2036.236895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2038.961747] Key type ._llcrypt registered [ 2038.966729] Key type .llcrypt registered [ 2039.178860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.194478] alg: No test for adler32 (adler32-zlib) [ 2040.166546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.279124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2040.304680] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2041.340723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2041.377769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2043.769793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2043.826208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2043.883265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2043.905990] LNetError: 109464:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2043.937763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2043.969758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2044.366692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2044.439642] LNet: 109484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.444951] LNetError: 109484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2044.450273] LNet: Removed LNI 10.1.2.3@tcp [ 2045.482993] LNet: Removed LNI 192.168.206.59@tcp [ 2046.131674] Key type .llcrypt unregistered [ 2046.134320] Key type ._llcrypt unregistered [ 2057.122791] Key type ._llcrypt registered [ 2057.124993] Key type .llcrypt registered [ 2058.425883] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:57:57 (1786733877) [ 2058.780963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.793784] alg: No test for adler32 (adler32-zlib) [ 2059.674642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.709721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.725757] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2060.288915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2060.361191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg659-client@tcp [ 2066.534455] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:58:05 (1786733885) [ 2066.666352] LNet: 111011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.682923] LNetError: 111011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.722196] LNet: Removed LNI 192.168.206.59@tcp [ 2067.597453] Key type .llcrypt unregistered [ 2067.603055] Key type ._llcrypt unregistered [ 2074.685748] Key type ._llcrypt registered [ 2074.691589] Key type .llcrypt registered [ 2075.015673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.054530] alg: No test for adler32 (adler32-zlib) [ 2076.021161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2076.057415] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2076.101480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2082.845927] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:58:21 (1786733901) [ 2084.326444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2086.233588] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:58:24 (1786733904) [ 2087.763875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2089.668436] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:58:28 (1786733908) [ 2089.749708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.775613] LNet: 112429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.782936] LNetError: 112429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.813104] LNet: Removed LNI 192.168.206.59@tcp [ 2089.854682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.948721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2090.000984] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2090.652746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2090.739455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2097.461289] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:58:36 (1786733916) [ 2097.601381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.657831] LNet: 113144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.665473] LNetError: 113144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.691893] LNet: Removed LNI 192.168.206.59@tcp [ 2097.734889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.815598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2097.857920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2098.691916] Key type .llcrypt unregistered [ 2098.699259] Key type ._llcrypt unregistered [ 2110.987150] Key type ._llcrypt registered [ 2110.990441] Key type .llcrypt registered [ 2112.212685] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:58:51 (1786733931) [ 2112.801351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2112.821526] alg: No test for adler32 (adler32-zlib) [ 2113.813374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.867640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.892103] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2114.435480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2114.489924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2121.221443] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:59:00 (1786733940) [ 2121.314658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.340885] LNet: 114697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.353780] LNetError: 114697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.384330] LNet: Removed LNI 192.168.206.59@tcp [ 2121.433969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.499964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2121.537354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2127.823734] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:59:06 (1786733946) [ 2128.019721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.059276] LNet: 115280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.088358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.162425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2128.537452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.568357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.660552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2129.231633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.275842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.383338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.414632] LNet: 115317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.436300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.525715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2129.882633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.906936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.948586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2130.320708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2130.343320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2136.760577] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:59:15 (1786733955) [ 2136.852348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.875119] LNet: 115911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.906119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.988798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.004820] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2137.594831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2137.615432] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2138.338156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2138.387619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2138.565741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.586526] LNetError: 116196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.591240] LNet: Removed LNI 192.168.206.59@tcp [ 2138.600529] LNet: Removed LNI 192.168.206.59@tcp2 [ 2138.633456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.694679] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2138.703717] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2138.711129] LNet: 116202:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2139.181832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2139.403225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.448325] LNet: Removed LNI 192.168.206.59@tcp [ 2140.452496] LNetError: 116239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.461383] LNetError: 116239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2140.499466] LNet: Removed LNI 192.168.206.59@tcp2 [ 2140.520124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.582322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.610854] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2141.311482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2141.345648] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2142.038379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2142.096322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2142.167770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2142.342850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.355553] LNet: 116524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.358517] LNet: 116524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2143.392345] LNet: Removed LNI 192.168.206.59@tcp [ 2143.396412] LNetError: 116524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.405989] LNetError: 116524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2143.430909] LNet: Removed LNI 192.168.206.59@tcp2 [ 2143.475246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.582772] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2143.593466] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2143.598142] LNet: 116530:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2144.135312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2144.329501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.350460] LNet: Removed LNI 192.168.206.59@tcp [ 2145.384022] LNet: Removed LNI 192.168.206.59@tcp2 [ 2145.432277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.500572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.526697] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2146.108846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2146.130213] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2146.807884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2146.860872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2147.030369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.064398] LNet: Removed LNI 192.168.206.59@tcp [ 2148.069731] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.084418] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2148.109906] LNet: Removed LNI 192.168.206.59@tcp2 [ 2148.173148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.272915] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2148.278789] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2148.285368] LNet: 116853:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2148.941964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2149.186229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.240918] LNet: Removed LNI 192.168.206.59@tcp [ 2150.262248] LNet: Removed LNI 192.168.206.59@tcp2 [ 2150.285902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.344262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.363220] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2150.929966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.952285] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2151.707955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2151.736382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2151.790433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2151.996705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.030283] LNet: 117174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.034817] LNet: 117174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2153.057123] LNet: Removed LNI 192.168.206.59@tcp [ 2154.100850] LNet: Removed LNI 192.168.206.59@tcp2 [ 2154.151770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.266820] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2154.271781] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2154.277391] LNet: 117181:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.827766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2154.976776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.990415] LNet: Removed LNI 192.168.206.59@tcp [ 2156.013787] LNet: Removed LNI 192.168.206.59@tcp2 [ 2156.047126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.112800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.133990] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2156.783201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2156.796593] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2157.464613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2157.509961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2157.551850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2157.824658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.838988] LNetError: 117515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.843710] LNetError: 117515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2158.881521] LNet: Removed LNI 192.168.206.59@tcp [ 2158.902671] LNet: Removed LNI 192.168.206.59@tcp2 [ 2158.955748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.037618] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2159.042559] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2159.046246] LNet: 117521:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2159.549677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2159.839141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.865063] LNet: Removed LNI 192.168.206.59@tcp [ 2160.876554] LNet: Removed LNI 192.168.206.59@tcp2 [ 2160.923375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.994796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.045789] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2161.803887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2161.838966] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2162.665388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2162.702975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2162.736633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2162.810321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2163.095357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.128260] LNet: Removed LNI 192.168.206.59@tcp [ 2165.160341] LNet: Removed LNI 192.168.206.59@tcp2 [ 2165.208883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.306082] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2165.313024] LNet: Added LNI 192.168.206.59@tcp2 [8/256/0/180] [ 2165.731239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2172.263968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2173.885682] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:59:52 (1786733992) [ 2173.957262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.980185] LNet: 118633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2173.985239] LNet: 118633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2174.001060] LNetError: 118633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.012564] LNetError: 118633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2174.030227] LNet: Removed LNI 192.168.206.59@tcp [ 2175.080156] LNet: Removed LNI 192.168.206.59@tcp2 [ 2175.112473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.428654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2175.445988] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2176.174482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2176.208609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2176.967212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2176.997655] LNet: Removed LNI 10.1.2.3@tcp [ 2177.008593] LNet: Removed LNI 192.168.206.59@tcp [ 2177.034195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.083422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2177.127949] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2177.132396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2177.556881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2177.585436] LNet: Removed LNI 192.168.206.59@tcp [ 2177.623976] LNet: Removed LNI 10.1.2.3@tcp [ 2183.351940] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:00:02 (1786734002) [ 2183.523122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.571449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.602581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.619781] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2184.191426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2184.218129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.245711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.273808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.322693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.379322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.446319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.510084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.569706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.635875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.687826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.59@tcp [ 2184.850815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2190.238262] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:00:09 (1786734009) [ 2190.387983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.469685] LNet: Removed LNI 192.168.206.59@tcp [ 2191.503171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.678640] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2191.876112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2191.883153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2191.889390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2194.197289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.234944] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2194.922449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2194.939160] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2196.389194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.59@tcp [ 2196.449875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2196.511919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.59@tcp -m PUT -e local_timeout -r 1 [ 2196.571805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2196.633817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2196.637752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2196.810828] LNetError: 120317:0:(lib-move.c:4801:lnet_parse()) 192.168.206.59@tcp, src 192.168.206.59@tcp: Dropping PUT (error -108 looking up sender) [ 2197.864678] LNet: Removed LNI 10.1.2.3@tcp [ 2198.892416] LNet: Removed LNI 192.168.206.59@tcp [ 2199.305526] Key type .llcrypt unregistered [ 2199.308509] Key type ._llcrypt unregistered [ 2208.622815] Key type ._llcrypt registered [ 2208.624555] Key type .llcrypt registered [ 2209.641631] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:00:28 (1786734028) [ 2210.131467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2210.159466] alg: No test for adler32 (adler32-zlib) [ 2211.034385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.341724] LNet: 121629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.829756] Key type .llcrypt unregistered [ 2211.831748] Key type ._llcrypt unregistered [ 2216.709981] Key type ._llcrypt registered [ 2216.712401] Key type .llcrypt registered [ 2216.981612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2216.995612] alg: No test for adler32 (adler32-zlib) [ 2217.905703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2217.931298] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2224.671854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 2224.908490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2225.539293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.159@tcp [ 2225.788539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2225.817979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2225.843523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2225.881819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2225.929041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2225.957633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2225.998470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2226.035448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2226.060510] LNet: 122077:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2226.083551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2226.139416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2226.145229] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.159@tcp [ 2226.178363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2226.204300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2226.241329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2226.274532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2226.849943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2227.091750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2227.124370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2227.149837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2227.172101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2227.200724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2227.230745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2227.254989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2227.284360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2227.299707] LNet: 122194:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2227.312221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2227.335952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2227.360884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2227.384916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2227.410909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2227.436769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2227.964428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.159@tcp [ 2228.251464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2228.285359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2228.317325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2228.358982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2228.394631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2228.420986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2228.450809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2228.477614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2228.493822] LNet: 122311:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2228.509368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2228.540269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2228.574701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2228.603219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2228.651345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2228.677836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2229.242078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2229.378499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2229.404327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2229.430193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2229.453388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2229.479404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2229.505675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2229.534526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2229.566672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2229.600846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2229.624900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2229.651509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2229.676797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2229.698887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2229.726675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2229.962382] LNet: 122475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.967454] LNetError: 122475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.999251] LNet: Removed LNI 192.168.206.59@tcp [ 2230.339578] Key type .llcrypt unregistered [ 2230.343358] Key type ._llcrypt unregistered [ 2244.856194] Key type ._llcrypt registered [ 2244.857757] Key type .llcrypt registered [ 2246.076293] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:01:05 (1786734065) [ 2246.407978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.422113] alg: No test for adler32 (adler32-zlib) [ 2247.366523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.750409] LNet: 123375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.263898] Key type .llcrypt unregistered [ 2248.269599] Key type ._llcrypt unregistered [ 2254.594511] Key type ._llcrypt registered [ 2254.596668] Key type .llcrypt registered [ 2254.872653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2254.901799] alg: No test for adler32 (adler32-zlib) [ 2255.763370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2255.781763] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2262.804739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.159@tcp [ 2271.892531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.956400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2271.976944] LNet: Added LNI 192.168.206.59@tcp1 [8/256/0/180] [ 2272.875443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2273.594575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.159@tcp [ 2273.824875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2273.885726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2273.940545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2273.981317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2274.013367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2274.042165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2274.074385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2274.103804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2274.122297] LNet: 124027:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2274.143955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2274.178709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2274.208802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2274.211385] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.159@tcp [ 2274.257624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2274.298401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2274.324042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.948327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2275.189491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2275.222557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2275.253070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2275.275949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2275.304869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2275.330140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2275.357473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2275.382338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2275.397802] LNet: 124145:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2275.417580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2275.441471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2275.471093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2275.496569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2275.525534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2275.559221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2276.190483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.159@tcp [ 2276.408332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2276.440315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2276.463751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2276.495811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2276.556326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2276.603932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2276.649928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2276.679619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2276.709879] LNet: 124262:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2276.728458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2276.751338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2276.789126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2276.817603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2276.843434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2276.885518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2277.564886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2277.806504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2277.833533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2277.865874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2277.896607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2277.920887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2277.947789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2277.975670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2278.007518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2278.051902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2278.097686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2278.147664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2278.189457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2278.229914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2278.294676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2283.317603] LNet: 124460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.330098] LNetError: 124460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.342530] LNet: Removed LNI 192.168.206.59@tcp [ 2283.354662] LNet: Removed LNI 192.168.206.59@tcp1 [ 2283.724068] Key type .llcrypt unregistered [ 2283.729080] Key type ._llcrypt unregistered [ 2299.582768] Key type ._llcrypt registered [ 2299.585222] Key type .llcrypt registered [ 2300.781551] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:01:59 (1786734119) [ 2301.998194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2303.660735] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:02:02 (1786734122) [ 2310.288410] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:02:09 (1786734129) [ 2317.293155] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:02:16 (1786734136) [ 2323.971467] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:02:22 (1786734142) [ 2330.592269] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:02:29 (1786734149) [ 2336.850049] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:02:36 (1786734156) [ 2343.971505] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:02:42 (1786734162) [ 2351.564929] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:02:50 (1786734170) [ 2357.972383] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:02:57 (1786734177) [ 2364.246296] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:03:03 (1786734183) [ 2370.792162] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:03:09 (1786734189) [ 2377.832188] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:03:16 (1786734196) [ 2384.926504] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:03:23 (1786734203) [ 2391.906348] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:03:30 (1786734210) [ 2400.244882] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:03:39 (1786734219) [ 2407.287695] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:03:46 (1786734226) [ 2414.122311] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:03:53 (1786734233) [ 2420.695802] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:03:59 (1786734239) [ 2421.054347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.063564] alg: No test for adler32 (adler32-zlib) [ 2421.992452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.112827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2422.132146] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2423.004805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.023281] LNet: 135460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.031268] LNetError: 135460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2423.041575] LNet: Removed LNI 192.168.206.59@tcp [ 2423.060295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.123741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2423.162117] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2429.143642] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:04:08 (1786734248) [ 2436.038775] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:04:15 (1786734255) [ 2444.191234] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:04:22 (1786734262) [ 2452.998966] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:04:31 (1786734271) [ 2460.033934] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:04:39 (1786734279) [ 2466.848716] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:04:45 (1786734285) [ 2473.952207] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:04:52 (1786734292) [ 2482.913383] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:05:01 (1786734301) [ 2490.430973] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:05:09 (1786734309) [ 2498.884252] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:05:17 (1786734317) [ 2504.519940] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:05:23 (1786734323) [ 2510.893223] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:05:30 (1786734330) [ 2517.229413] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2298 sec ========== 15:05:36 (1786734336) [ 2517.487679] LNet: 142929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.497145] LNetError: 142929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2517.525090] LNet: Removed LNI 192.168.206.59@tcp [ 2518.009483] Key type .llcrypt unregistered [ 2518.013431] Key type ._llcrypt unregistered [ 2528.070833] Key type ._llcrypt registered [ 2528.074872] Key type .llcrypt registered [ 2528.275792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2528.283639] alg: No test for adler32 (adler32-zlib) [ 2529.284053] Lustre: Lustre: Build Version: 2.17.56_2_gd9f03f2 [ 2529.478676] LNet: Added LNI 192.168.206.59@tcp [8/256/0/180] [ 2531.136944] Key type lgssc registered [ 2532.219507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2584.487222] Lustre: Mounted lustre-client [ 2589.192306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2596.943248] Lustre: Unmounted lustre-client [ 2633.120726] Key type lgssc unregistered [ 2633.318696] LNet: 145856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2633.331275] LNetError: 145856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2633.346086] LNet: Removed LNI 192.168.206.59@tcp [ 2633.887513] Key type .llcrypt unregistered [ 2633.891780] Key type ._llcrypt unregistered