[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 438230974 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002296] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007583] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.011077] LSM: Security Framework initializing [ 0.012041] Yama: becoming mindful. [ 0.013028] SELinux: Initializing. [ 0.014058] *** VALIDATE selinux *** [ 0.021742] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026099] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027130] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.031422] *** VALIDATE proc *** [ 0.033160] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036127] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038024] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042325] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.044753] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045589] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046018] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053284] rcu: Hierarchical SRCU implementation. [ 0.055298] smp: Bringing up secondary CPUs ... [ 0.056480] x86: Booting SMP configuration: [ 0.057016] .... node #0, CPUs: #1 #2 #3 [ 0.067100] smp: Brought up 1 node, 4 CPUs [ 0.069009] smpboot: Max logical packages: 1 [ 0.070014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.127621] node 0 deferred pages initialised in 55ms [ 0.132115] devtmpfs: initialized [ 0.133278] x86/mm: Memory block size: 128MB [ 0.135735] gcov: version magic: 0x41383552 [ 0.137250] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.138087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139276] pinctrl core: initialized pinctrl subsystem [ 0.140210] [ 0.140810] ************************************************************* [ 0.141062] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.142024] ** ** [ 0.143025] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.144011] ** ** [ 0.145022] ** This means that this kernel is built to expose internal ** [ 0.146010] ** IOMMU data structures, which may compromise security on ** [ 0.147018] ** your system. ** [ 0.148030] ** ** [ 0.149014] ** If you see this message and you are not debugging the ** [ 0.150018] ** kernel, report this immediately to your vendor! ** [ 0.151025] ** ** [ 0.152025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153009] ************************************************************* [ 0.154859] NET: Registered protocol family 16 [ 0.155442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159093] cpuidle: using governor menu [ 0.161963] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165524] PCI: Using configuration type 1 for base access [ 0.168156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184078] cryptd: max_cpu_qlen set to 1000 [ 0.187231] ACPI: Added _OSI(Module Device) [ 0.188009] ACPI: Added _OSI(Processor Device) [ 0.189012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194181] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197476] ACPI: Interpreter enabled [ 0.198046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199010] ACPI: Using IOAPIC for interrupt routing [ 0.200125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201369] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.210732] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221471] acpiphp: Slot [2] registered [ 0.223165] acpiphp: Slot [5] registered [ 0.224087] acpiphp: Slot [6] registered [ 0.225075] acpiphp: Slot [3] registered [ 0.227134] acpiphp: Slot [4] registered [ 0.228082] acpiphp: Slot [7] registered [ 0.229068] acpiphp: Slot [8] registered [ 0.230102] acpiphp: Slot [9] registered [ 0.231093] acpiphp: Slot [10] registered [ 0.233084] acpiphp: Slot [11] registered [ 0.234067] acpiphp: Slot [12] registered [ 0.235073] acpiphp: Slot [13] registered [ 0.236067] acpiphp: Slot [14] registered [ 0.238075] acpiphp: Slot [15] registered [ 0.239078] acpiphp: Slot [16] registered [ 0.240068] acpiphp: Slot [17] registered [ 0.241082] acpiphp: Slot [18] registered [ 0.242127] acpiphp: Slot [19] registered [ 0.243092] acpiphp: Slot [20] registered [ 0.245058] acpiphp: Slot [21] registered [ 0.246073] acpiphp: Slot [22] registered [ 0.247084] acpiphp: Slot [23] registered [ 0.248135] acpiphp: Slot [24] registered [ 0.249063] acpiphp: Slot [25] registered [ 0.250075] acpiphp: Slot [26] registered [ 0.252222] acpiphp: Slot [27] registered [ 0.253064] acpiphp: Slot [28] registered [ 0.254059] acpiphp: Slot [29] registered [ 0.256072] acpiphp: Slot [30] registered [ 0.257077] acpiphp: Slot [31] registered [ 0.258090] PCI host bridge to bus 0000:00 [ 0.259013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.261014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.263013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.265013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.267015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.270017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.271124] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.274118] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.277125] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.284012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.288056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.291014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.293012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.295012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297512] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301387] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.303041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.306610] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.310011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.320012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.326027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.330561] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.340012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.349000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.360012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.369576] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.385012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.429014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.449375] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.451274] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.453244] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.455265] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.458175] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.462164] iommu: Default domain type: Passthrough [ 0.465108] SCSI subsystem initialized [ 0.466149] ACPI: bus type USB registered [ 0.467089] usbcore: registered new interface driver usbfs [ 0.469066] usbcore: registered new interface driver hub [ 0.470065] usbcore: registered new device driver usb [ 0.472184] pps_core: LinuxPPS API ver. 1 registered [ 0.473010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.476117] PTP clock support registered [ 0.478071] EDAC MC: Ver: 3.0.0 [ 0.480177] PCI: Using ACPI for IRQ routing [ 0.481720] NetLabel: Initializing [ 0.483016] NetLabel: domain hash size = 128 [ 0.484006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.485057] NetLabel: unlabeled traffic allowed by default [ 0.487263] vgaarb: loaded [ 0.488281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.490010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.498498] clocksource: Switched to clocksource kvm-clock [ 0.600491] VFS: Disk quotas dquot_6.6.0 [ 0.601654] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.603556] *** VALIDATE ramfs *** [ 0.604332] *** VALIDATE hugetlbfs *** [ 0.605294] pnp: PnP ACPI init [ 0.607904] pnp: PnP ACPI: found 6 devices [ 0.622374] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.625157] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.626755] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.628512] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.630503] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.632481] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.635050] NET: Registered protocol family 2 [ 0.637096] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.641351] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.644918] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.649089] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.652170] TCP: Hash tables configured (established 65536 bind 65536) [ 0.654517] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.657248] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.659702] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.662288] NET: Registered protocol family 1 [ 0.665324] RPC: Registered named UNIX socket transport module. [ 0.667351] RPC: Registered udp transport module. [ 0.668967] RPC: Registered tcp transport module. [ 0.670704] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.673837] NET: Registered protocol family 44 [ 0.675548] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.677583] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.679618] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.681783] PCI: CLS 0 bytes, default 64 [ 0.683370] Unpacking initramfs... [ 2.054846] debug: unmapping init [mem 0xffff9c353cc64000-0xffff9c353ffcffff] [ 2.059681] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.062207] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.065406] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.596739] Initialise system trusted keyrings [ 2.598676] Key type blacklist registered [ 2.600832] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.610642] zbud: loaded [ 2.614143] *** VALIDATE nfs *** [ 2.615866] *** VALIDATE nfs4 *** [ 2.617897] pstore: using deflate compression [ 2.624256] Platform Keyring initialized [ 2.762883] NET: Registered protocol family 38 [ 2.764542] Key type asymmetric registered [ 2.765986] Asymmetric key parser 'x509' registered [ 2.767339] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.770140] io scheduler mq-deadline registered [ 2.771642] io scheduler kyber registered [ 2.773178] io scheduler bfq registered [ 2.775888] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.778482] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.781154] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.783833] ACPI: Power Button [PWRF] [ 2.788703] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.797207] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.836914] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.863455] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.893466] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.897872] Non-volatile memory driver v1.3 [ 2.899510] Linux agpgart interface v0.103 [ 2.927841] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.932340] vda: detected capacity change from 0 to 74686464 [ 2.954701] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.957919] vdb: detected capacity change from 0 to 1073741824 [ 2.971714] libphy: Fixed MDIO Bus: probed [ 2.977190] usbcore: registered new interface driver usbserial_generic [ 2.979526] usbserial: USB Serial support registered for generic [ 2.982901] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.987977] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.989975] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.992317] mousedev: PS/2 mouse device common for all mice [ 2.995353] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.995927] rtc_cmos 00:05: RTC can wake from S4 [ 3.003160] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.007933] rtc_cmos 00:05: registered as rtc0 [ 3.010605] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.012350] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.019094] intel_pstate: CPU model not supported [ 3.022756] hid: raw HID events driver (C) Jiri Kosina [ 3.025242] usbcore: registered new interface driver usbhid [ 3.027200] usbhid: USB HID core driver [ 3.028663] drop_monitor: Initializing network drop monitor service [ 3.031106] Initializing XFRM netlink socket [ 3.033344] NET: Registered protocol family 10 [ 3.036415] Segment Routing with IPv6 [ 3.038039] NET: Registered protocol family 17 [ 3.040037] mpls_gso: MPLS GSO support [ 3.046041] RAS: Correctable Errors collector initialized. [ 3.048081] AVX version of gcm_enc/dec engaged. [ 3.049729] AES CTR mode by8 optimization enabled [ 3.121322] sched_clock: Marking stable (3121283399, 0)->(4083546877, -962263478) [ 3.124751] registered taskstats version 1 [ 3.126350] Loading compiled-in X.509 certificates [ 3.128035] zswap: loaded using pool lzo/zbud [ 3.151063] Key type big_key registered [ 3.161763] Key type encrypted registered [ 3.163194] ima: No TPM chip found, activating TPM-bypass! [ 3.165349] ima: Allocated hash algorithm: sha1 [ 3.167534] ima: No architecture policies found [ 3.169430] evm: Initialising EVM extended attributes: [ 3.171540] evm: security.selinux [ 3.173081] evm: security.ima [ 3.174323] evm: security.capability [ 3.175584] evm: HMAC attrs: 0x1 [ 3.178155] rtc_cmos 00:05: setting system clock to 2026-08-03 19:46:02 UTC (1785786362) [ 3.185668] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.189405] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.199090] Write protecting the kernel read-only data: 28672k [ 3.202799] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.205326] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.239303] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.246955] systemd[1]: Detected virtualization kvm. [ 3.248862] systemd[1]: Detected architecture x86-64. [ 3.250624] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.276169] systemd[1]: No hostname configured. [ 3.277528] systemd[1]: Set hostname to . [ 3.279310] random: systemd: uninitialized urandom read (16 bytes read) [ 3.281461] systemd[1]: Initializing machine ID from random generator. [ 3.330643] random: ln: uninitialized urandom read (6 bytes read) [ 3.407341] random: systemd: uninitialized urandom read (16 bytes read) [ 3.409595] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.415316] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.418862] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.037887] device-mapper: uevent: version 1.0.3 [ 4.040227] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 4.325800] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.749627] virtio_net virtio0 ens2: renamed from eth0 [ 4.815425] scsi host0: ata_piix [ 4.844461] scsi host1: ata_piix [ 4.847496] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.850257] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.210693] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.680787] random: crng init done [ 9.682227] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.087754] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.233835] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.581561] SELinux: Disabled at runtime. [ 11.646993] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.657166] systemd[1]: Detected virtualization kvm. [ 11.659257] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.182967] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.186287] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.191135] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.194886] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.198227] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.210190] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.216818] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [[ 12.271264] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.676362] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.968208] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.999572] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.070833] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.091905] EDAC sbridge: Ver: 1.1.2 [ 14.280303] Key type dns_resolver registered [ 14.620980] NFS: Registering the id_resolver key type [ 14.622707] Key type id_resolver registered [ 14.624141] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg409-client login: [ 109.827120] libcfs: loading out-of-tree module taints kernel. [ 110.241915] Key type ._llcrypt registered [ 110.249921] Key type .llcrypt registered [ 110.918767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 110.962867] alg: No test for adler32 (adler32-zlib) [ 112.860068] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 114.263876] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 115.967148] Key type lgssc registered [ 119.752854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 197.666019] hrtimer: interrupt took 10972442 ns [ 306.719765] Lustre: Mounted lustre-client [ 313.257458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 329.114746] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing check_logdir /tmp/testlogs/ [ 332.255308] Lustre: lustre-OST0000-osc-ffff9c3584be4000: disconnect after 22s idle [ 336.695476] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing yml_node [ 342.636648] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 345.943476] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 349.508884] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 351.482413] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 3 15:51:49 EDT 2026 [ 370.975743] Lustre: Unmounted lustre-client [ 406.193944] Key type lgssc unregistered [ 406.766345] LNet: 8401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.775983] LNetError: 8401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 406.822996] LNet: Removed LNI 192.168.204.9@tcp [ 408.186291] Key type .llcrypt unregistered [ 408.188151] Key type ._llcrypt unregistered [ 443.054251] Key type ._llcrypt registered [ 443.056500] Key type .llcrypt registered [ 443.225888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.238613] alg: No test for adler32 (adler32-zlib) [ 444.132185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 444.216425] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 444.245256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 450.426550] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 450.544951] LNet: 9278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.570781] LNetError: 9278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.607135] LNet: Removed LNI 192.168.204.9@tcp [ 451.627490] Key type .llcrypt unregistered [ 451.635081] Key type ._llcrypt unregistered [ 464.028582] Key type ._llcrypt registered [ 464.038800] Key type .llcrypt registered [ 466.173434] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:53:43 (1785786823) [ 467.098188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 467.121691] alg: No test for adler32 (adler32-zlib) [ 468.358303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.596141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.665558] LNet: 9699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.709370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.822267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.855781] LNet: 9706:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 480.612164] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:53:58 (1785786838) [ 481.186492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.236404] LNet: 10286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.282472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.411623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 491.932326] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:54:08 (1785786848) [ 493.177447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.301592] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.407364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.743908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 505.565110] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:54:22 (1785786862) [ 505.771483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.817315] LNet: 11450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.851572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.999789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 518.309350] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:54:35 (1785786875) [ 519.237926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.307436] LNet: 12041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.344541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.433138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 520.256080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 530.451183] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:54:47 (1785786887) [ 530.804358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.842261] LNet: 12633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.856119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.953151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 531.857050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 533.092365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 543.449248] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:55:00 (1785786900) [ 543.632717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.754838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.957040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 555.849546] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:55:12 (1785786912) [ 556.021470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.090088] LNet: 13809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.103701] LNet: 13809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 556.160663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.374352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 556.444139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 557.449850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 557.560931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 558.601039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 558.751070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 559.806712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 559.863956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 560.735361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 560.826477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 561.427524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 561.516127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 562.100229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 562.162877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 562.772712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 562.825872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 563.873123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 563.990435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 565.140700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 565.184550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 565.619868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 565.648910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 565.941685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 565.967317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 566.338575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 566.362972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 577.340655] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:55:34 (1785786934) [ 577.477173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.519141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.613828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 577.706885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 589.947966] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:55:47 (1785786947) [ 590.081335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.129077] LNet: 15078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.138813] LNet: 15078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 590.185809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.357469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 590.434087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 603.112616] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:55:59 (1785786959) [ 603.598709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.733522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.910349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 603.969107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 616.468089] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:56:13 (1785786973) [ 617.396873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.507669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.754304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 617.881631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 627.167843] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:56:24 (1785786984) [ 627.282672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.375432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.594717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 627.709750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 637.594478] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:56:35 (1785786995) [ 638.642987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.771670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.966732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 639.075556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 649.307650] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:56:46 (1785787006) [ 649.453912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.559775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.769388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 650.525329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 660.787289] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:56:57 (1785787017) [ 660.917359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.969083] LNet: 18609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.979419] LNet: 18609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 661.022199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.353219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 661.763675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 672.891160] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:57:09 (1785787029) [ 673.074169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.154506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.294996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.556537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 682.383308] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:57:20 (1785787040) [ 682.727552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.761040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.333685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.715133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 693.812747] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:57:30 (1785787050) [ 693.962446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.047308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.236980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.300987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 711.521746] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:57:47 (1785787067) [ 711.804775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.939144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.212755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.368581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 728.144717] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:58:05 (1785787085) [ 728.302887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.373749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.582450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 739.864566] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:58:17 (1785787097) [ 740.130195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.272609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.534591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 750.354908] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:58:27 (1785787107) [ 750.489305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.608332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.873364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 751.926534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 762.226357] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:58:39 (1785787119) [ 762.669323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.740899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.858375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 762.926646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 772.672391] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:58:50 (1785787130) [ 772.885736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.962396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.151225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 773.201764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 783.308229] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:59:00 (1785787140) [ 783.435470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.500548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.567891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 783.604789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 793.780457] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:59:10 (1785787150) [ 794.231165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.263480] LNet: 25233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.271918] LNet: 25233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 794.313581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.462992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 794.530984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 805.055772] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:59:21 (1785787161) [ 805.644399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.725182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.846192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 805.965322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 805.995919] LNetError: 25833:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 806.057680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 806.111413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 814.788721] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:59:32 (1785787172) [ 815.206412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.267024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.420180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LvdORAgVCl [ 815.535287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LvdORAgVCl [ 815.623542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LvdORAgVCl [ 823.108517] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:59:40 (1785787180) [ 823.453402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.532968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.655742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 823.749239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 823.837994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 832.896288] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:59:50 (1785787190) [ 833.199783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.224593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.269585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.283451] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 834.932935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 837.234503] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 839.431583] Key type lgssc registered [ 841.384770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.270978] LNetError: 27596:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.9@tcp99 has no available nets [ 851.502301] Key type lgssc unregistered [ 851.997337] LNetError: 28705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.027383] LNet: Removed LNI 192.168.204.9@tcp [ 852.949147] Key type .llcrypt unregistered [ 852.956452] Key type ._llcrypt unregistered [ 863.224259] Key type ._llcrypt registered [ 863.230700] Key type .llcrypt registered [ 864.916379] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:00:22 (1785787222) [ 865.456568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.476244] alg: No test for adler32 (adler32-zlib) [ 866.530723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.625126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 871.566748] LNet: Message forwarding will be enabled in 2s [ 874.655219] LNet: Message forwarding enabled [ 875.409429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 875.465638] LNet: Message forwarding will stop in 2s to 6s [ 879.217364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 881.391261] LNet: Message forwarding disabled [ 882.823083] LNet: Message forwarding will be enabled in 2s [ 886.562314] LNet: Message forwarding enabled [ 891.577981] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:00:49 (1785787249) [ 891.765527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.820631] LNet: 30104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.805390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.908049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 900.651545] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:00:58 (1785787258) [ 901.049945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.112869] LNet: 30689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.153568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.222913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 901.247552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 901.286845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 901.317264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 901.344618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 901.378448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 901.401538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 901.437913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 901.500601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 901.569402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 901.623111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 901.650136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 901.686300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 901.720227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 901.751485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 901.790397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 901.820696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 901.858872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 901.891504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 901.911897] LNetError: 30732:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 910.841475] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:01:07 (1785787267) [ 910.974190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.017269] LNet: 31309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.047067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.203813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 919.712927] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:01:17 (1785787277) [ 922.962686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg409-server [ 925.359499] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:01:23 (1785787283) [ 925.926975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.983341] LNet: 32248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.007714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.078953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.108136] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 926.890628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[10-14]@tcp [ 926.914766] LNet: 32381:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 926.968632] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 926.988408] LNetError: 32252:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.12@tcp has gone from up to down [ 927.973471] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 927.988978] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 929.013705] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 929.026386] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 930.032620] LNetError: 32256:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.10@tcp has gone from up to down [ 930.045166] LNetError: 32256:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 931.125270] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 931.143169] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 935.208464] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 935.226039] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 937.275417] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:01:34 (1785787294) [ 940.141133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg409-server [ 942.698207] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:01:39 (1785787299) [ 942.995579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.398225] LNetError: 32251:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.204.12@tcp [ 978.207547] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 979.935219] LNetError: 32382:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Error -11 reading HELLO from 192.168.204.13 [ 979.938772] LNetError: Unexpected error -11 connecting to 192.168.204.11@tcp at host 192.168.204.11:988 [ 979.951159] LNetError: 32382:0:(socklnd_cb.c:1675:ksocknal_recv_hello()) Skipped 1 previous similar message [ 979.976098] LNetError: Skipped 1 previous similar message [ 980.960643] LNetError: 32252:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.11@tcp has gone from up to down [ 980.977946] LNet: 33317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.992680] LNetError: 33317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.031223] LNet: Removed LNI 192.168.204.9@tcp [ 981.076383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.160188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.213094] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 981.952304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[10-15/2]@tcp [ 982.010463] LNet: 33440:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 982.018194] LNet: 33440:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 982.052820] LNetError: 33327:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 982.070361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[10-15/2]@tcp [ 982.075481] LNetError: 33327:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 990.583599] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:02:28 (1785787348) [ 990.834554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.888660] LNet: 34031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.923400] LNetError: 34031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.957273] LNet: Removed LNI 192.168.204.9@tcp [ 991.001687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.172858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 991.265646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.324097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.417138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 991.538934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.653041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.730592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 991.869490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.924683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.002253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 992.107520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.168175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.233771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 992.305687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.364838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.441757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 992.540436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.578212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.630538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 992.727843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.784749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.903995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 993.084800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.121894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.198115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.304476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.361288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.449864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 993.552793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.603614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.662064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 993.749279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.806671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.895919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1000.979685] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:02:38 (1785787358) [ 1003.677240] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg409-server [ 1005.944532] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:02:43 (1785787363) [ 1008.236835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg409-server [ 1010.455792] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:02:48 (1785787368) [ 1010.781662] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.791552] LNet: 35434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1011.529520] Key type .llcrypt unregistered [ 1011.534303] Key type ._llcrypt unregistered [ 1019.922992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1020.042853] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1020.053711] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1020.055836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1022.704321] Key type ._llcrypt registered [ 1022.708628] Key type .llcrypt registered [ 1022.914162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.941439] alg: No test for adler32 (adler32-zlib) [ 1023.898164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.953502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1023.995995] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1024.819249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1024.842996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1027.325199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1027.353906] LNet: Removed LNI 10.1.2.3@tcp [ 1027.541168] LNet: 36155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.549567] LNetError: 36155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.585542] LNet: Removed LNI 192.168.204.9@tcp [ 1028.752521] Key type .llcrypt unregistered [ 1028.756779] Key type ._llcrypt unregistered [ 1045.249265] Key type ._llcrypt registered [ 1045.251524] Key type .llcrypt registered [ 1047.050316] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:03:24 (1785787404) [ 1048.061414] Key type .llcrypt unregistered [ 1048.069174] Key type ._llcrypt unregistered [ 1057.056659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1057.426492] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1057.449642] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1057.463321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1060.376294] Key type ._llcrypt registered [ 1060.383155] Key type .llcrypt registered [ 1060.668383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.703252] alg: No test for adler32 (adler32-zlib) [ 1061.855559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.049398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.177252] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1063.315558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1063.341439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.463267] LNet: 37648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.488956] LNetError: 37648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.501160] LNet: Removed LNI 10.1.2.3@tcp [ 1068.532318] LNet: Removed LNI 192.168.204.9@tcp [ 1069.708053] Key type .llcrypt unregistered [ 1069.719927] Key type ._llcrypt unregistered [ 1083.677495] Key type ._llcrypt registered [ 1083.682200] Key type .llcrypt registered [ 1085.219696] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:04:02 (1785787442) [ 1085.853244] Key type .llcrypt unregistered [ 1085.857065] Key type ._llcrypt unregistered [ 1094.122658] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1094.290979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1094.301437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1094.311741] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1096.171077] Key type ._llcrypt registered [ 1096.174186] Key type .llcrypt registered [ 1096.503906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.547026] alg: No test for adler32 (adler32-zlib) [ 1097.543560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1097.625558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1097.663847] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1100.124324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1100.151840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1102.683930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1102.703888] LNet: Removed LNI 10.1.2.3@tcp [ 1102.717291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1102.754729] LNet: Removed LNI 10.1.2.31@tcp [ 1102.769530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1102.791818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1103.654568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1103.723927] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1104.798477] LNet: 39563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.808559] LNetError: 39563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 1104.811241] LNet: Removed LNI 10.1.2.31@tcp [ 1104.829228] LNet: Removed LNI 10.1.2.3@tcp [ 1105.599700] Key type .llcrypt unregistered [ 1105.602475] Key type ._llcrypt unregistered [ 1121.563632] Key type ._llcrypt registered [ 1121.565158] Key type .llcrypt registered [ 1123.344582] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:04:40 (1785787480) [ 1123.953392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.996600] alg: No test for adler32 (adler32-zlib) [ 1125.009887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.121387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1125.164690] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1126.418375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1126.453164] LNet: 40517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.463980] LNetError: 40517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.491215] LNet: Removed LNI 192.168.204.9@tcp [ 1126.530959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.617381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.9@tcp [ 1126.660437] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1127.917145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.9@tcp [ 1127.971738] LNet: Removed LNI 192.168.204.9@tcp [ 1128.138518] LNet: 40652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.323599] Key type .llcrypt unregistered [ 1129.330246] Key type ._llcrypt unregistered [ 1145.141189] Key type ._llcrypt registered [ 1145.148021] Key type .llcrypt registered [ 1147.199510] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:05:04 (1785787504) [ 1150.296423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg409-server [ 1153.998858] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:05:10 (1785787510) [ 1154.613529] Key type .llcrypt unregistered [ 1154.614921] Key type ._llcrypt unregistered [ 1164.212812] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1164.480912] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1164.497025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1164.526689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1167.357234] Key type ._llcrypt registered [ 1167.363454] Key type .llcrypt registered [ 1167.610781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.653747] alg: No test for adler32 (adler32-zlib) [ 1168.651305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.743470] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1169.813962] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1172.409958] LNet: 42491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.423943] LNetError: 42491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.435203] LNet: Removed LNI 192.168.204.9@tcp1 [ 1172.457715] LNet: Removed LNI 10.1.2.3@tcp2 [ 1173.178610] Key type .llcrypt unregistered [ 1173.182735] Key type ._llcrypt unregistered [ 1190.966332] Key type ._llcrypt registered [ 1190.968340] Key type .llcrypt registered [ 1192.345328] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:05:50 (1785787550) [ 1193.051665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.086754] alg: No test for adler32 (adler32-zlib) [ 1194.056758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.097688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1194.112579] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1194.731947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1194.752744] LNet: 43462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.761968] LNetError: 43462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.787100] LNet: Removed LNI 192.168.204.9@tcp [ 1194.850675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.960630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1194.990575] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1203.655895] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:06:00 (1785787560) [ 1203.850872] LNet: 44049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.867217] LNetError: 44049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.889330] LNet: Removed LNI 192.168.204.9@tcp [ 1204.998730] Key type .llcrypt unregistered [ 1205.010321] Key type ._llcrypt unregistered [ 1214.359520] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1214.592698] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1214.601709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1214.612090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1217.419426] Key type ._llcrypt registered [ 1217.424482] Key type .llcrypt registered [ 1217.756893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.779617] alg: No test for adler32 (adler32-zlib) [ 1218.785738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.901619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1218.967622] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1220.163712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1220.224971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1223.067106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1223.077931] LNet: 44798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.081121] LNetError: 44798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1223.086286] LNet: Removed LNI 10.1.2.3@tcp [ 1223.094255] LNet: Removed LNI 192.168.204.9@tcp [ 1223.108176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1223.140396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1223.153782] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1223.156051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1223.385691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1223.396844] LNet: Removed LNI 10.1.2.3@tcp [ 1223.407526] LNet: Removed LNI 192.168.204.9@tcp [ 1223.422128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1223.485241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1223.519211] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1223.526316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1223.984623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1223.998377] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1225.150636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1225.167311] LNet: 44987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.170479] LNet: 44987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1225.175719] LNetError: 44987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1225.180536] LNetError: 44987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1225.185858] LNet: Removed LNI 10.1.2.3@tcp [ 1225.188570] LNet: Removed LNI 192.168.204.9@tcp [ 1225.197741] LNet: Removed LNI 192.168.204.9@tcp2 [ 1225.250081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.332228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1225.374624] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1225.380352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1225.386991] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 1235.123491] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:06:32 (1785787592) [ 1235.489293] LNet: 45584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.505942] LNetError: 45584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1235.518285] LNetError: 45584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1235.537142] LNet: Removed LNI 10.1.2.3@tcp [ 1235.541756] LNet: Removed LNI 192.168.204.9@tcp [ 1235.565929] LNet: Removed LNI 192.168.204.9@tcp2 [ 1236.523652] Key type .llcrypt unregistered [ 1236.535144] Key type ._llcrypt unregistered [ 1245.902386] Key type ._llcrypt registered [ 1245.903855] Key type .llcrypt registered [ 1246.171953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1246.196178] alg: No test for adler32 (adler32-zlib) [ 1247.150714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.209903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1247.240064] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1248.463297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1248.516270] LNet: 45980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.532296] LNetError: 45980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.563351] LNet: Removed LNI 192.168.204.9@tcp [ 1248.609658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.705153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1248.743875] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1249.531636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1249.565727] LNet: 46010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.574933] LNetError: 46010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.600347] LNet: Removed LNI 192.168.204.9@tcp [ 1249.651832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.757201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1249.796217] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1250.608019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.625744] LNet: 46040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.629680] LNetError: 46040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.642855] LNet: Removed LNI 192.168.204.9@tcp [ 1250.684298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.793154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1250.836351] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1251.923781] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1252.235134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1252.252166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1252.261966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1253.767532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.830194] LNet: 46109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.833800] LNetError: 46109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.842841] LNet: Removed LNI 192.168.204.9@tcp [ 1253.921205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.997435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1254.037639] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1263.359844] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:06:59 (1785787619) [ 1263.527922] LNet: 46698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.542202] LNetError: 46698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.589753] LNet: Removed LNI 192.168.204.9@tcp [ 1264.910546] Key type .llcrypt unregistered [ 1264.922137] Key type ._llcrypt unregistered [ 1273.572611] Key type ._llcrypt registered [ 1273.575706] Key type .llcrypt registered [ 1274.040478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1274.069066] alg: No test for adler32 (adler32-zlib) [ 1274.997278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.058482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1275.082292] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1275.959431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1275.974511] LNet: 47106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.985064] LNetError: 47106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.004644] LNet: Removed LNI 192.168.204.9@tcp [ 1276.032755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.111303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1276.154468] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1286.087820] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:07:23 (1785787643) [ 1286.539823] LNet: 47698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.558222] LNetError: 47698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.594712] LNet: Removed LNI 192.168.204.9@tcp [ 1287.406418] Key type .llcrypt unregistered [ 1287.408491] Key type ._llcrypt unregistered [ 1294.859857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1295.116561] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1295.125905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1295.151490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1297.851327] Key type ._llcrypt registered [ 1297.854649] Key type .llcrypt registered [ 1298.037764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1298.062119] alg: No test for adler32 (adler32-zlib) [ 1299.081228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.180840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1299.222871] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1300.421495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.458924] LNet: 48148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.474719] LNetError: 48148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.498368] LNet: Removed LNI 192.168.204.9@tcp [ 1300.565837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.679659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1300.729324] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1309.665789] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:07:47 (1785787667) [ 1309.867285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1309.920492] LNet: 48743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.932500] LNetError: 48743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.986239] LNet: Removed LNI 192.168.204.9@tcp [ 1310.022639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.156870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1318.376257] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:07:55 (1785787675) [ 1318.541339] LNet: 49321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.511675] Key type .llcrypt unregistered [ 1319.518104] Key type ._llcrypt unregistered [ 1328.398893] Key type ._llcrypt registered [ 1328.405922] Key type .llcrypt registered [ 1328.734299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.775165] alg: No test for adler32 (adler32-zlib) [ 1329.784534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.835136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.857089] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1331.009963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1331.049405] LNet: 49728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.060836] LNetError: 49728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.082792] LNet: Removed LNI 192.168.204.9@tcp [ 1331.116742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1331.211906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1331.245784] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1341.218494] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:08:18 (1785787698) [ 1341.328996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1341.375050] LNet: 50315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.387574] LNetError: 50315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.423534] LNet: Removed LNI 192.168.204.9@tcp [ 1341.513488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.685322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1352.393245] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:08:29 (1785787709) [ 1354.948427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1358.593959] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:08:34 (1785787714) [ 1362.632317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1367.622637] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:08:42 (1785787722) [ 1370.812633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1374.670592] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:08:50 (1785787730) [ 1378.970095] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1382.946132] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:08:59 (1785787739) [ 1387.056958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1391.524534] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:09:06 (1785787746) [ 1391.938171] LNet: 52374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.313179] Key type .llcrypt unregistered [ 1393.316256] Key type ._llcrypt unregistered [ 1406.326493] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1406.662266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1406.690268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1406.699416] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1409.600936] Key type ._llcrypt registered [ 1409.602367] Key type .llcrypt registered [ 1409.980845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.031681] alg: No test for adler32 (adler32-zlib) [ 1411.098767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.206251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1411.246811] LNet: Added LNI 192.168.204.9@tcp [8/512/0/180] [ 1411.259342] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1411.260896] LNet: 52700:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1411.395159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.434913] LNet: 52719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.447682] LNetError: 52719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1411.459957] LNet: Removed LNI 10.1.2.3@tcp [ 1411.467687] LNetError: 52719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.505507] LNet: Removed LNI 192.168.204.9@tcp [ 1411.537389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.612498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1411.652484] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1411.737335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.758226] LNet: 52743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.768820] LNetError: 52743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.792857] LNet: Removed LNI 192.168.204.9@tcp [ 1411.837102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.939439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1411.976204] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1411.981920] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1411.983587] LNet: 52751:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1412.129278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.165463] LNet: 52769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.177686] LNetError: 52769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1412.182991] LNet: Removed LNI 10.1.2.3@tcp [ 1412.184874] LNetError: 52769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.205745] LNet: Removed LNI 192.168.204.9@tcp [ 1412.237914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.360713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1412.395766] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1412.413739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1412.421225] LNet: 52777:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1412.710541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.750439] LNet: 52803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.771428] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1412.779391] LNet: Removed LNI 10.1.2.3@tcp [ 1412.781748] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.826210] LNet: Removed LNI 192.168.204.9@tcp [ 1412.883538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.026728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1413.070685] LNet: Added LNI 192.168.204.9@tcp [8/256/0/182] [ 1413.078333] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1413.284759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.335149] LNet: 52831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.348247] LNetError: 52831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1413.357756] LNet: Removed LNI 10.1.2.3@tcp [ 1413.359074] LNetError: 52831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.398410] LNet: Removed LNI 192.168.204.9@tcp [ 1413.458184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.555666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1413.606486] LNet: Added LNI 192.168.204.9@tcp [8/256/0/182] [ 1413.616754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1413.622745] LNet: 52839:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1413.726239] LNet: Removed LNI 10.1.2.3@tcp [ 1413.750793] LNet: Removed LNI 192.168.204.9@tcp [ 1414.978862] Key type .llcrypt unregistered [ 1414.983444] Key type ._llcrypt unregistered [ 1436.648407] Key type ._llcrypt registered [ 1436.652873] Key type .llcrypt registered [ 1438.803057] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:09:55 (1785787795) [ 1439.382954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.419700] alg: No test for adler32 (adler32-zlib) [ 1440.349631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.386809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1440.402800] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1441.246656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.9 [ 1441.266972] LNet: 53818:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1441.274634] LNet: Removed LNI 192.168.204.9@tcp [ 1449.478056] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:10:06 (1785787806) [ 1449.878081] LNet: 54390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.889734] LNetError: 54390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.930554] LNet: Removed LNI 192.168.204.9@tcp [ 1450.672289] Key type .llcrypt unregistered [ 1450.674158] Key type ._llcrypt unregistered [ 1460.005178] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1460.230906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1463.240238] Key type ._llcrypt registered [ 1463.243515] Key type .llcrypt registered [ 1463.603178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.652323] alg: No test for adler32 (adler32-zlib) [ 1465.474660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1476.243981] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:10:33 (1785787833) [ 1476.927228] LNet: 55304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.945651] LNetError: 55304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1476.983355] LNet: Removed LNI 10.1.2.3@tcp [ 1478.217631] Key type .llcrypt unregistered [ 1478.221368] Key type ._llcrypt unregistered [ 1488.801809] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1488.995444] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1488.998447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1489.002537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1491.999493] Key type ._llcrypt registered [ 1492.007593] Key type .llcrypt registered [ 1492.266730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1492.282848] alg: No test for adler32 (adler32-zlib) [ 1493.371673] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1502.757909] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:10:59 (1785787859) [ 1504.970637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1507.874571] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:11:04 (1785787864) [ 1509.259661] LNet: 56492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.273571] LNetError: 56492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1509.305433] LNet: Removed LNI 10.1.2.3@tcp [ 1510.549481] Key type .llcrypt unregistered [ 1510.553064] Key type ._llcrypt unregistered [ 1521.532075] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1521.816359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1521.827269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1521.838392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1523.912916] Key type ._llcrypt registered [ 1523.915610] Key type .llcrypt registered [ 1524.280632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1524.324497] alg: No test for adler32 (adler32-zlib) [ 1525.318684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.436449] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1535.735436] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:11:33 (1785787893) [ 1536.340223] LNet: 57714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.376874] LNetError: 57714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1536.392936] LNet: Removed LNI 10.1.2.3@tcp [ 1537.466581] Key type .llcrypt unregistered [ 1537.468844] Key type ._llcrypt unregistered [ 1545.411941] Key type ._llcrypt registered [ 1545.424691] Key type .llcrypt registered [ 1545.738776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1545.785410] alg: No test for adler32 (adler32-zlib) [ 1546.928042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1547.006292] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1556.546786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1557.551416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1557.613089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1557.657908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1557.671604] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1557.837504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1557.859616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1557.882594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1557.914751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1557.945783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1557.982996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1558.026284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1558.083755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1558.109212] LNet: 58154:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1558.130739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1558.203274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1558.267435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1558.304435] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1558.321717] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1558.341097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1558.393421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1558.453306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1558.515485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1559.232706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1559.287169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1559.334339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1559.377082] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1559.842766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1559.915640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1559.979788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1560.006475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1560.060695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1560.116715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1560.166387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1560.242928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1560.297353] LNet: 58263:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.339422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1560.415375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1560.496429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1560.542330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1560.591398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.684567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1560.749834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1561.364230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1561.446945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1561.502894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1561.540295] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1561.785583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1561.866394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1561.916919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1561.981245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1562.102979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1562.187588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1562.258553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1562.321050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1562.358515] LNet: 58373:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1562.419761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1562.506728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1562.583377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1562.665040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1562.750542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1562.842355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1562.949478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1563.986991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1564.058529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1564.103445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1564.394959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1564.424761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1564.482740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1564.550369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1564.611340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1564.667715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1564.734494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1564.823627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1564.863409] LNet: 58482:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1564.904374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1564.985162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1565.075911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1565.129481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1565.191436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1565.252896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1565.326810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1566.502384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1566.543047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1566.614726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1566.650694] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1566.653601] LNet: Skipped 1 previous similar message [ 1566.973088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1567.035778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1567.106584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1567.158945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1567.210872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1567.255951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1567.337085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1567.385064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1567.464840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1567.560365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1567.650034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1567.738414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1567.829571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1567.912100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1568.020492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1569.109412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1569.172815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1569.237094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1569.698979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1569.743371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1569.801111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1569.859218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1569.907563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1569.934530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1569.978347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1570.012674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1570.040215] LNet: 58701:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1570.044762] LNet: 58701:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1570.070191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1570.115350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1570.159461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1570.187841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1570.209508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1570.229465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1570.256320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1570.547291] LNet: 58742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.560110] LNetError: 58742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.589779] LNet: Removed LNI 192.168.204.9@tcp [ 1571.061042] Key type .llcrypt unregistered [ 1571.062603] Key type ._llcrypt unregistered [ 1597.245834] Key type ._llcrypt registered [ 1597.247507] Key type .llcrypt registered [ 1599.329128] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:12:36 (1785787956) [ 1601.215225] Key type .llcrypt unregistered [ 1601.217318] Key type ._llcrypt unregistered [ 1611.123807] Key type ._llcrypt registered [ 1611.127266] Key type .llcrypt registered [ 1611.414893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.453676] alg: No test for adler32 (adler32-zlib) [ 1612.593397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1612.647589] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1624.065611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1641.996029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.078857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1642.109159] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1643.900734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1643.976359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 1 [ 1644.082513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1644.208189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 1 [ 1644.430048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_interrupt -r 1 [ 1644.469065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 1 [ 1644.518861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_interrupt -r 1 [ 1644.574704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 1 [ 1644.637320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1644.676649] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1644.686163] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1645.044539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1645.120883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1645.224834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1645.301742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1645.368197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1645.467724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1645.534876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1645.599861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1645.622261] LNet: 60214:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1645.660992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1645.704698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1645.737110] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1645.753868] LNet: Skipped 3 previous similar messages [ 1645.767667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1645.818814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1645.885235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1645.940740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.991045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1646.936145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1647.023678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 1 [ 1647.092244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1647.164818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 1 [ 1647.214653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_dropped -r 1 [ 1647.277276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 1 [ 1647.376029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_dropped -r 1 [ 1647.440974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 1 [ 1647.509978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1647.543547] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1647.926735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1647.978929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1648.026857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.064141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.099785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.133394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1648.187635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.240435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1648.260159] LNet: 60336:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1648.282470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1648.340888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1648.393648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1648.436754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1648.477669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1648.517821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1648.575201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1649.058678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1649.114991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 1 [ 1649.181767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1649.239986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 1 [ 1649.303156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_aborted -r 1 [ 1649.359714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 1 [ 1649.420556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_aborted -r 1 [ 1649.476859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 1 [ 1649.542069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1649.589921] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1649.597918] LNet: Skipped 2 previous similar messages [ 1649.985605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1650.049463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1650.120665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1650.200870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.261895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.325735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.380599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.444449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.477752] LNet: 60457:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.523770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.584232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.653582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.730752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.782533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.839668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1650.891835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1651.495638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1651.531741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 1 [ 1651.592945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1651.641972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 1 [ 1651.689581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_no_route -r 1 [ 1651.752803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 1 [ 1651.807418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_no_route -r 1 [ 1651.861702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 1 [ 1651.928866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1652.287782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1652.328786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1652.370551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1652.416765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1652.475555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.551926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.608889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.672216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.695344] LNet: 60578:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1652.745916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.803426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.885912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.936980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1653.020628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1653.069762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1653.151861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1654.136573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1654.221532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 1 [ 1654.289694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1654.342906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 1 [ 1654.379312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_timeout -r 1 [ 1654.414386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 1 [ 1654.454268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_timeout -r 1 [ 1654.490036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 1 [ 1654.526452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1654.546776] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1654.552089] LNet: Skipped 7 previous similar messages [ 1654.811375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1654.862538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1654.904630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1654.937217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1654.974668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1655.019838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1655.084399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1655.134064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1655.189787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1655.230420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1655.292687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1655.334713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1655.387685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1655.450556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1655.520234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1656.267615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1656.313342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_error -r 1 [ 1656.351445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1656.402784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_error -r 1 [ 1656.459083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_error -r 1 [ 1656.521531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_error -r 1 [ 1656.585622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_error -r 1 [ 1656.647536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_error -r 1 [ 1656.701426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1657.024821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.077936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1657.123644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1657.164994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1657.220169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1657.276464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1657.333361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1657.386853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1657.411156] LNet: 60821:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.421343] LNet: 60821:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1657.449324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1657.505503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1657.548640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1657.588574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1657.641768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1657.696497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1657.745807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1665.004861] LNet: 60897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.018235] LNetError: 60897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.031630] LNet: Removed LNI 192.168.204.9@tcp [ 1666.110472] LNet: Removed LNI 192.168.204.9@tcp1 [ 1666.884555] Key type .llcrypt unregistered [ 1666.888778] Key type ._llcrypt unregistered [ 1690.608882] Key type ._llcrypt registered [ 1690.612893] Key type .llcrypt registered [ 1692.292455] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:14:09 (1785788049) [ 1693.114724] Key type .llcrypt unregistered [ 1693.119562] Key type ._llcrypt unregistered [ 1701.952274] Key type ._llcrypt registered [ 1701.959061] Key type .llcrypt registered [ 1702.451501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1702.492547] alg: No test for adler32 (adler32-zlib) [ 1703.753704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1703.808571] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1713.303670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1714.036848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1714.089934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1714.152522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1714.176420] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1714.457911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1714.502878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1714.556904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1714.600465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1714.674845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1714.746489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1714.810186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.874412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.929433] LNet: 62150:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.970594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1715.047304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1715.129449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1715.253735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1715.316168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1715.407538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1715.475714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1716.351019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1716.399382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1716.467635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1716.499992] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1716.820685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1716.856843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1716.907589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1716.936704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.972124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1717.040473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1717.089340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1717.156488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1717.184501] LNet: 62259:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.213586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1717.265267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1717.304138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1717.359331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1717.407985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1717.467469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.504833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1718.158074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1718.215751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1718.268595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1718.306165] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1718.730526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.787882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.838757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.874174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.912309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.960325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.996731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.046674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.079950] LNet: 62369:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.129830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.191218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.248985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1719.299191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1719.353924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1719.394732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.450232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1719.976808] LNet: 62410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.992066] LNetError: 62410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.016925] LNet: Removed LNI 192.168.204.9@tcp [ 1720.591819] Key type .llcrypt unregistered [ 1720.595283] Key type ._llcrypt unregistered [ 1744.328898] Key type ._llcrypt registered [ 1744.335045] Key type .llcrypt registered [ 1746.322893] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:15:03 (1785788103) [ 1747.451315] Key type .llcrypt unregistered [ 1747.452932] Key type ._llcrypt unregistered [ 1757.709197] Key type ._llcrypt registered [ 1757.727152] Key type .llcrypt registered [ 1758.158885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1758.205453] alg: No test for adler32 (adler32-zlib) [ 1759.258889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1759.297962] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1769.780807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1782.610359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.709259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1782.751487] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1785.006314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1785.066369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 1 [ 1785.142345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1785.201284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 1 [ 1785.271781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_dropped -r 1 [ 1785.332487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 1 [ 1785.390910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_dropped -r 1 [ 1785.467314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 1 [ 1785.542177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1785.580961] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1785.594215] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1786.039893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1786.130293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1786.246735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1786.338180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1786.425784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1786.504434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1786.575259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1786.657996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1786.660451] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1786.675628] LNet: Skipped 3 previous similar messages [ 1786.700363] LNet: 63879:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1786.742345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1786.797795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1786.852091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1786.953758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1787.038596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1787.132914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1787.218887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1787.287775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1788.156359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1788.244456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_error -r 1 [ 1788.348695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1788.423053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_error -r 1 [ 1788.496780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_error -r 1 [ 1788.585351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_error -r 1 [ 1788.697851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_error -r 1 [ 1788.791409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_error -r 1 [ 1788.868212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1788.894909] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1789.249228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.327019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.420535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.535996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.627537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.776790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1789.885319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1789.984230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.026876] LNet: 64003:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.073478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.133289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.245554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.311475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.393888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.481814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.569500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1790.675168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.242844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1792.337564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_timeout -r 1 [ 1792.432401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1792.518110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_timeout -r 1 [ 1792.633162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_timeout -r 1 [ 1792.785291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_timeout -r 1 [ 1792.897198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_timeout -r 1 [ 1793.000430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_timeout -r 1 [ 1793.102833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.109@tcp [ 1793.140518] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1793.518602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1793.695487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1793.807097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1793.977763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1794.069858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1794.146961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1794.243729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.323252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1794.415602] LNet: 64126:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1794.461270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1794.580393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1794.689172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1794.838628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1794.971913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1795.073416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1795.160405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1795.282335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1802.874678] LNet: 64205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.913937] LNetError: 64205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.929472] LNet: Removed LNI 192.168.204.9@tcp [ 1802.955264] LNet: Removed LNI 192.168.204.9@tcp1 [ 1803.786682] Key type .llcrypt unregistered [ 1803.798886] Key type ._llcrypt unregistered [ 1831.356301] Key type ._llcrypt registered [ 1831.358962] Key type .llcrypt registered [ 1833.067386] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:16:30 (1785788190) [ 1835.287231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1837.081780] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:16:35 (1785788195) [ 1837.994278] Key type .llcrypt unregistered [ 1837.997606] Key type ._llcrypt unregistered [ 1846.363993] Key type ._llcrypt registered [ 1846.369110] Key type .llcrypt registered [ 1846.689989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1846.713422] alg: No test for adler32 (adler32-zlib) [ 1847.789077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1847.848104] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1857.268131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1857.959663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1857.995090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1858.068530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1858.116221] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1858.147650] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1858.356439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1858.383361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1858.408514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1858.432313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1858.460659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1858.488972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1858.521662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1858.551936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1858.569379] LNet: 65752:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1858.587142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1858.628852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1858.666440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1858.707503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1858.741428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.791519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1859.041618] LNet: 65791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.046658] LNetError: 65791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.061580] LNet: Removed LNI 192.168.204.9@tcp [ 1859.544487] Key type .llcrypt unregistered [ 1859.546084] Key type ._llcrypt unregistered [ 1882.674244] Key type ._llcrypt registered [ 1882.677876] Key type .llcrypt registered [ 1882.874469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1882.907871] alg: No test for adler32 (adler32-zlib) [ 1883.902442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1883.937631] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1893.976780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1902.603698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.674482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1902.722860] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1905.621151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1905.687748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e network_timeout -r 1 [ 1905.766689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1905.886881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e network_timeout -r 1 [ 1905.976849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e network_timeout -r 1 [ 1906.056754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e network_timeout -r 1 [ 1906.171785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e network_timeout -r 1 [ 1906.271952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e network_timeout -r 1 [ 1906.351126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 1906.394248] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1906.463445] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1906.465488] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 1906.825954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1906.914237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1906.961826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1906.991711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1907.028488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1907.061092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1907.125929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1907.205251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1907.224574] LNet: 66672:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1907.265129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1907.325547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1907.374290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1907.427675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1907.497088] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1907.498778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1907.576351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1914.696467] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 1914.713794] LNet: Skipped 1 previous similar message [ 1915.163041] LNet: 66746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.178261] LNetError: 66746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.191222] LNet: Removed LNI 192.168.204.9@tcp [ 1916.199092] LNetError: 66746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.227243] LNet: Removed LNI 192.168.204.9@tcp1 [ 1917.065820] Key type .llcrypt unregistered [ 1917.067910] Key type ._llcrypt unregistered [ 1939.640439] Key type ._llcrypt registered [ 1939.644611] Key type .llcrypt registered [ 1941.172582] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:18:18 (1785788298) [ 1942.130544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.201559] alg: No test for adler32 (adler32-zlib) [ 1943.163219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.214419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.234109] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1944.092421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1944.117511] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1945.441710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1945.535672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1945.568846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1945.610713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1945.643949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1945.676848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1945.715141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1945.750868] LNetError: 67847:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1946.273589] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 1946.274677] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1948.319947] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1949.488784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1952.415433] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1953.165804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1953.222773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1953.311143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.347663] LNet: 67954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.356728] LNetError: 67954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1953.365094] LNet: Removed LNI 192.168.204.9@tcp [ 1953.375992] LNet: Removed LNI 192.168.204.9@tcp1 [ 1953.420775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.509849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1953.578317] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1954.646178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1954.670674] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1955.844230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1955.903390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1956.025280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1956.102161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1956.175315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1956.219240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1956.268518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1956.316108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 1956.350445] LNetError: 68226:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 1956.511993] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1958.842797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1962.655297] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1962.670607] LNet: Skipped 1 previous similar message [ 1967.096126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1967.165313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1967.211553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1977.027312] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:18:54 (1785788334) [ 1977.277511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.328330] LNet: 68931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.347952] LNetError: 68931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.357606] LNetError: 68931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1977.368404] LNet: Removed LNI 192.168.204.9@tcp [ 1977.402295] LNet: Removed LNI 192.168.204.9@tcp1 [ 1977.464108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.568220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.610898] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 1978.680307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1978.709680] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 1980.010435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 1980.148793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1980.210689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1980.256023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1980.311515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1980.397657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1980.460144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 0 [ 1980.580841] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.9@tcp [ 1980.616562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1981.599535] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 1981.612199] LNet: Skipped 1 previous similar message [ 1996.510332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1996.744604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1996.805752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1996.856933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 500 [ 1997.021644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2002.792465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2002.860778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.886807] LNet: 69418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.907514] LNetError: 69418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.915145] LNetError: 69418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2002.925512] LNet: Removed LNI 192.168.204.9@tcp [ 2002.942919] LNet: Removed LNI 192.168.204.9@tcp1 [ 2003.010596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.118578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.165426] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2004.288707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2004.348671] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2005.676894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2005.777975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2005.887720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2005.931140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2005.998567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2006.061243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2006.111497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2006.165217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.9@tcp --health 0 [ 2007.135379] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 2007.142415] LNet: Skipped 3 previous similar messages [ 2007.621099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2018.122820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2018.177498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2018.245304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2028.771412] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:19:45 (1785788385) [ 2030.120112] LNet: 70465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.140716] LNetError: 70465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.157410] LNetError: 70465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2030.186124] LNet: Removed LNI 192.168.204.9@tcp [ 2030.217397] LNet: Removed LNI 192.168.204.9@tcp1 [ 2031.454964] Key type .llcrypt unregistered [ 2031.458889] Key type ._llcrypt unregistered [ 2044.229068] Key type ._llcrypt registered [ 2044.230992] Key type .llcrypt registered [ 2044.776653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2044.858473] alg: No test for adler32 (adler32-zlib) [ 2046.235993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2046.331579] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2068.186810] Lustre: DEBUG MARKER: Initial discovery [ 2068.707274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 2072.017406] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 2072.095230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 2072.183820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 2072.227941] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2072.272322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2072.340809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.109@tcp -r 1 -m GET -l 3 [ 2072.398839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.109@tcp -r 1 -m GET -e local_error [ 2072.471155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.109@tcp -r 1 -m PUT -l 6 [ 2072.991194] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 2074.613716] Lustre: DEBUG MARKER: Force oleg409-client.virtnet to discover 192.168.204.109@tcp (in background) [ 2074.972781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.109@tcp [ 2076.661356] Lustre: DEBUG MARKER: Force oleg409-server to discover 192.168.204.9@tcp [ 2078.047841] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2080.122730] Lustre: DEBUG MARKER: Wait for 71234 [ 2082.258737] Lustre: DEBUG MARKER: Finished wait on 71234 [ 2082.317545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 2082.380437] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2082.385402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2082.736283] LNet: 71682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.749072] LNetError: 71682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.772118] LNet: Removed LNI 192.168.204.9@tcp [ 2083.563778] Key type .llcrypt unregistered [ 2083.568393] Key type ._llcrypt unregistered [ 2104.976758] Key type ._llcrypt registered [ 2104.984272] Key type .llcrypt registered [ 2106.633937] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:21:04 (1785788464) [ 2107.521328] Key type .llcrypt unregistered [ 2107.525171] Key type ._llcrypt unregistered [ 2116.284132] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2116.572871] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2116.581502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2116.597616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2119.557052] Key type ._llcrypt registered [ 2119.558592] Key type .llcrypt registered [ 2119.940218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.986718] alg: No test for adler32 (adler32-zlib) [ 2121.067203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.128936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2121.162746] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2121.854729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2121.917279] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2137.506243] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:21:34 (1785788494) [ 2137.865278] LNet: 73756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.892563] LNetError: 73756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2137.905866] LNet: Removed LNI 10.1.2.3@tcp [ 2137.944921] LNet: Removed LNI 192.168.204.9@tcp [ 2138.880196] Key type .llcrypt unregistered [ 2138.889191] Key type ._llcrypt unregistered [ 2148.971264] Key type ._llcrypt registered [ 2148.976987] Key type .llcrypt registered [ 2149.408384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2149.442713] alg: No test for adler32 (adler32-zlib) [ 2150.423718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.489363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.522756] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2151.664639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2151.709701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2153.882184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.9@tcp [ 2154.989897] LNetError: Unexpected error -22 connecting to 192.168.204.9@tcp at host 192.168.204.9:988 [ 2164.984853] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:22:02 (1785788522) [ 2165.168537] LNet: 74884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.180785] LNetError: 74884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2165.194834] LNet: Removed LNI 10.1.2.3@tcp [ 2165.265306] LNet: Removed LNI 192.168.204.9@tcp [ 2166.523978] Key type .llcrypt unregistered [ 2166.528197] Key type ._llcrypt unregistered [ 2175.573964] Key type ._llcrypt registered [ 2175.579246] Key type .llcrypt registered [ 2175.975915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2176.009038] alg: No test for adler32 (adler32-zlib) [ 2177.089242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.174370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2177.221698] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2178.483640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2178.519599] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 2179.533967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.9@tcp1 --nid 192.168.204.9@tcp2 [ 2185.647814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp1 192.168.204.9@tcp2 [ 2185.663648] LNetError: 75727:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.9@tcp1 to 192.168.204.9@tcp2 - Route table may be misconfigured [ 2185.689080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp2 192.168.204.9@tcp1 [ 2196.772903] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:22:33 (1785788553) [ 2196.898397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.940211] LNet: 76295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.948767] LNetError: 76295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.967187] LNet: Removed LNI 192.168.204.9@tcp1 [ 2197.031512] LNet: Removed LNI 192.168.204.9@tcp2 [ 2197.154060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.307524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2197.371363] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2199.308046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2199.351696] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2201.196452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2201.300285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.9@tcp [ 2201.366498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp1 192.168.204.9@tcp1 [ 2201.439474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp -d 192.168.204.9@tcp -e network_timeout [ 2201.538557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -e network_timeout [ 2201.654029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -e network_timeout [ 2201.756468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -e network_timeout [ 2201.831068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2201.867651] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 2201.908410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2202.184379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2202.404269] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 2202.421725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2213.338171] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:22:49 (1785788569) [ 2213.601773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.676097] LNet: 77157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.722121] LNetError: 77157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.740613] LNetError: 77157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2213.771911] LNet: Removed LNI 192.168.204.9@tcp [ 2214.822804] LNet: Removed LNI 192.168.204.9@tcp1 [ 2214.916452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.179080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2215.828710] LNet: 77201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.368447] Key type .llcrypt unregistered [ 2216.372489] Key type ._llcrypt unregistered [ 2231.000227] Key type ._llcrypt registered [ 2231.001879] Key type .llcrypt registered [ 2232.590202] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:23:10 (1785788590) [ 2233.534504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.587130] alg: No test for adler32 (adler32-zlib) [ 2234.678568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.530854] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2239.399560] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:23:16 (1785788596) [ 2239.712743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.767149] LNet: 78299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.796734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.948822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.005136] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2240.880157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2240.923706] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2242.448376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2242.572345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp1 [ 2242.649276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp1 [ 2251.447145] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:23:28 (1785788608) [ 2253.358340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg409-server [ 2255.347645] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:23:33 (1785788613) [ 2257.356288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg409-server [ 2260.352943] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:23:37 (1785788617) [ 2262.885372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg409-server [ 2265.773833] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:23:42 (1785788622) [ 2269.074306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg409-server [ 2271.771309] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:23:48 (1785788628) [ 2275.193171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg409-server [ 2277.968817] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:23:55 (1785788635) [ 2281.079774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg409-server [ 2283.994788] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:24:01 (1785788641) [ 2287.152827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg409-server [ 2289.915303] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:24:07 (1785788647) [ 2292.860928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg409-server [ 2294.432783] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:24:12 (1785788652) [ 2297.162351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg409-server [ 2299.820938] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:24:17 (1785788657) [ 2299.979845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.018825] LNet: 82180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.040701] LNetError: 82180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.051805] LNet: Removed LNI 192.168.204.9@tcp [ 2300.074210] LNet: Removed LNI 192.168.204.9@tcp1 [ 2300.141511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.282158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.345214] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2301.256430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2301.354580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2301.553645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.589415] LNet: 82330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.618920] LNetError: 82330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.628315] LNetError: 82330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2302.693776] LNet: Removed LNI 192.168.204.9@tcp [ 2302.742377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.873393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2302.948210] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2304.162688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2304.400887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.425791] LNet: 82475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.434910] LNetError: 82475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.454435] LNet: Removed LNI 192.168.204.9@tcp [ 2304.492806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.604772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.677013] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2305.406671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2305.510291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2305.793594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.925056] LNet: Removed LNI 192.168.204.9@tcp [ 2306.969878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.097768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2307.147704] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2308.396671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2308.757182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.825570] LNet: 82768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.840952] LNet: 82768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2308.864829] LNetError: 82768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.881186] LNetError: 82768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2308.910723] LNet: Removed LNI 192.168.204.9@tcp [ 2308.991864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.137816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2309.200243] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2310.368291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2310.426627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2311.894814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.939681] LNet: Removed LNI 192.168.204.9@tcp [ 2312.992541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.091415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2313.145202] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2314.223464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2314.516099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.545505] LNetError: 83070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.553239] LNetError: 83070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2314.578697] LNet: Removed LNI 192.168.204.9@tcp [ 2314.601986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.686611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2314.726193] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2315.477808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2315.580589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2315.845402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.909668] LNet: Removed LNI 192.168.204.9@tcp [ 2316.954901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.078146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2317.146575] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2318.570552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2320.343734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.386210] LNet: 83372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.394068] LNet: 83372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2321.450696] LNet: Removed LNI 192.168.204.9@tcp [ 2321.492726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.587548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2321.627358] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2322.673751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2322.769793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2323.177143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.202493] LNetError: 83520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.209569] LNetError: 83520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2324.266690] LNet: Removed LNI 192.168.204.9@tcp [ 2324.317614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.425521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2324.455775] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2325.640287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2325.991161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.083200] LNet: Removed LNI 192.168.204.9@tcp [ 2327.133888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2327.208930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2327.255379] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2328.573014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2328.733378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2330.290530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.379838] LNet: Removed LNI 192.168.204.9@tcp [ 2331.443316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.520437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2331.555348] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2332.670632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2333.057510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.125072] LNet: Removed LNI 192.168.204.9@tcp [ 2334.175790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.264897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.320179] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2334.975250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2335.087334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.168152] LNet: Removed LNI 192.168.204.9@tcp [ 2335.197797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.300210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.339520] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2336.736744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2336.862976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.896984] LNet: 84238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.905523] LNet: 84238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2336.937622] LNet: Removed LNI 192.168.204.9@tcp [ 2336.988510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.074277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2337.234131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.296276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.389372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2337.530937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.588882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.676408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2337.821639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.871273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.946912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2337.990786] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2339.168651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.223686] LNetError: 84408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.229842] LNetError: 84408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2339.271704] LNet: Removed LNI 192.168.204.9@tcp [ 2339.298533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.476178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2348.037753] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:25:05 (1785788705) [ 2348.164408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.239888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.367914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2348.426324] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 2350.427400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.457735] LNet: Removed LNI 192.168.204.9@tcp231 [ 2350.491353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.590272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2350.627747] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 2350.632172] LNet: 85136:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.209735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2351.251749] LNet: Removed LNI 192.168.204.9@tcp231 [ 2351.284041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2351.343106] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 2353.183332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.247616] LNet: Removed LNI 192.168.204.9@tcp231 [ 2353.279503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.415480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2353.464659] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/0] [ 2353.486096] LNet: 85289:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.428386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2354.504791] LNet: Removed LNI 192.168.204.9@tcp231 [ 2354.543822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2354.626516] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/0] [ 2356.702107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.735239] LNet: Removed LNI 192.168.204.9@tcp231 [ 2356.792283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.885721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2356.924161] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/60] [ 2356.931826] LNet: 85441:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.479647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2357.533961] LNet: Removed LNI 192.168.204.9@tcp231 [ 2357.563264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2357.597649] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/60] [ 2359.387554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.416641] LNet: Removed LNI 192.168.204.9@tcp231 [ 2359.469287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.518419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2359.539115] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 2359.544559] LNet: 85592:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.894317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2359.926678] LNet: Removed LNI 192.168.204.9@tcp231 [ 2359.961337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2360.000427] LNet: Added LNI 192.168.204.9@tcp231 [8/256/0/180] [ 2374.204992] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:25:31 (1785788731) [ 2374.878853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.991163] LNet: 86305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.997290] LNet: 86305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 2375.009996] LNetError: 86305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2375.027430] LNetError: 86305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2375.060425] LNet: Removed LNI 192.168.204.9@tcp231 [ 2375.109806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.297562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2375.369547] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 2376.636592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2376.711268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2376.762711] LNet: Removed LNI 192.168.204.9@tcp232 [ 2376.782730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2376.820560] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 2378.074603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2378.176270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2378.231199] LNet: Removed LNI 192.168.204.9@tcp232 [ 2378.245884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2378.277542] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 2379.006285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2379.039199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2379.104125] LNet: Removed LNI 192.168.204.9@tcp232 [ 2379.132491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2379.185852] LNet: Added LNI 192.168.204.9@tcp232 [8/256/0/180] [ 2380.486124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2380.549380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2380.595141] LNet: Removed LNI 192.168.204.9@tcp232 [ 2380.614416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2390.534173] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:25:47 (1785788747) [ 2392.255116] Key type .llcrypt unregistered [ 2392.258458] Key type ._llcrypt unregistered [ 2401.247811] Key type ._llcrypt registered [ 2401.249605] Key type .llcrypt registered [ 2401.667671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2401.705908] alg: No test for adler32 (adler32-zlib) [ 2402.779917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2402.820761] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2412.426305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 2426.808025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.876552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2426.929681] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2430.010735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_interrupt -r 2 [ 2430.090503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 2 [ 2430.168305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_interrupt -r 2 [ 2430.270299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 2 [ 2430.334913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_interrupt -r 2 [ 2430.421438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_interrupt -r 2 [ 2430.513820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_interrupt -r 2 [ 2430.582654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_interrupt -r 2 [ 2430.669813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 2430.724451] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2430.727612] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 2431.349772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2431.429907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2431.496425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2431.618252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2431.710482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2431.814131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.884705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.930326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.956558] LNet: 88100:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.993823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2432.048567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2432.080419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2432.116853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2432.151707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2432.176567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.706907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_dropped -r 2 [ 2432.747495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 2 [ 2432.787835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_dropped -r 2 [ 2432.839801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 2 [ 2432.873296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_dropped -r 2 [ 2432.937474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_dropped -r 2 [ 2432.979565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_dropped -r 2 [ 2433.019188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_dropped -r 2 [ 2433.070481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 2433.097776] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2433.411980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.476265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.515548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.557433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.586899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.609079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.641602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.671670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.687498] LNet: 88220:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.701835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.729261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.757407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.780852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.806550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.840893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.343363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_aborted -r 2 [ 2434.375385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 2 [ 2434.411702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_aborted -r 2 [ 2434.456637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 2 [ 2434.540573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_aborted -r 2 [ 2434.600646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_aborted -r 2 [ 2434.637573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_aborted -r 2 [ 2434.671036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_aborted -r 2 [ 2434.711648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 2434.731101] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2434.988301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2435.019062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2435.043497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2435.070567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2435.104974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2435.132885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2435.159492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2435.190370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.205566] LNet: 88340:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2435.219489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2435.245312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2435.270530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.301686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2435.332153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2435.363941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.914174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_no_route -r 2 [ 2435.953556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 2 [ 2435.998772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_no_route -r 2 [ 2436.076292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 2 [ 2436.117733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_no_route -r 2 [ 2436.164378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_no_route -r 2 [ 2436.213685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_no_route -r 2 [ 2436.270825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_no_route -r 2 [ 2436.318298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 2436.700577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.760450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2436.821851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2436.859538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2436.920694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2436.983429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2437.044256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2437.097596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2437.141500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2437.173510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2437.207687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2437.229623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2437.268615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2437.287842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2437.926284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e local_timeout -r 2 [ 2437.966623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 2 [ 2438.003817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e local_timeout -r 2 [ 2438.033792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 2 [ 2438.068236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e local_timeout -r 2 [ 2438.096364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e local_timeout -r 2 [ 2438.146416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e local_timeout -r 2 [ 2438.176178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e local_timeout -r 2 [ 2438.201418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 2438.218917] LNet: There was an unexpected network error while writing to 192.168.204.109: rc = -22 [ 2438.222689] LNet: Skipped 1 previous similar message [ 2438.467795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2438.489662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2438.518550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2438.552593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2438.580462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2438.606605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2438.635683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2438.663489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2438.678804] LNet: 88581:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2438.683985] LNet: 88581:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2438.697322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2438.727539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2438.779905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2438.837496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2438.881429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2438.919774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2439.504509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp -m GET -e remote_dropped -r 2 [ 2439.531633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 2 [ 2439.571224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp -m GET -e remote_dropped -r 2 [ 2439.605088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 2 [ 2439.633604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp -m GET -e remote_dropped -r 2 [ 2439.662795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.9@tcp1 -d 192.168.204.109@tcp1 -m GET -e remote_dropped -r 2 [ 2439.700606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp -m GET -e remote_dropped -r 2 [ 2439.731735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.9@tcp1 -d 192.168.204.9@tcp1 -m GET -e remote_dropped -r 2 [ 2439.765660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.9@tcp 192.168.204.109@tcp [ 2439.802519] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 2440.069428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2440.097640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2440.128322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2440.167452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2440.208630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2440.257210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2440.307616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2440.341282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2440.385322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2440.423975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2440.459966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2440.489451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2440.513939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2440.536223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2447.213872] LNet: 88775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.219178] LNetError: 88775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2447.229279] LNet: Removed LNI 192.168.204.9@tcp [ 2447.239777] LNet: Removed LNI 192.168.204.9@tcp1 [ 2447.781546] Key type .llcrypt unregistered [ 2447.786084] Key type ._llcrypt unregistered [ 2467.611070] Key type ._llcrypt registered [ 2467.612862] Key type .llcrypt registered [ 2468.837568] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:27:06 (1785788826) [ 2470.141473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2470.151418] alg: No test for adler32 (adler32-zlib) [ 2471.047948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2471.092938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2471.117270] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2472.358214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2472.386209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.9@tcp [ 2472.418064] LNetError: 89754:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 2472.562813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.9@tcp [ 2472.744533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2482.187482] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:27:19 (1785788839) [ 2482.317702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2482.344590] LNet: 90340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2482.359442] LNetError: 90340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2482.375104] LNet: Removed LNI 192.168.204.9@tcp [ 2482.407142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.482327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2482.507050] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2483.164168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2483.179081] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2483.918255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.9@tcp [ 2483.939842] LNetError: 90612:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.9@tcp to recovery [ 2484.511317] LNet: 1 local NIs in recovery (showing 1): 192.168.204.9@tcp [ 2484.515286] LNet: There was an unexpected network error while writing to 192.168.204.9: rc = -22 [ 2485.211266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.9@tcp [ 2495.265528] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:27:32 (1785788852) [ 2497.730772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg409-server [ 2500.117921] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:27:37 (1785788857) [ 2503.200965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg409-server [ 2505.336792] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:27:42 (1785788862) [ 2508.012987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg409-server [ 2510.339460] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:27:47 (1785788867) [ 2510.457783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2510.510700] LNet: 92238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.530637] LNetError: 92238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.540821] LNet: Removed LNI 192.168.204.9@tcp [ 2510.561267] LNet: Removed LNI 192.168.204.9@tcp1 [ 2510.634484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.742728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2510.784719] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2511.858722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2511.963563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2512.019781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2512.145691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2512.177412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2512.211564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2520.433218] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:27:57 (1785788877) [ 2520.587129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2520.665554] LNet: 92963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.684781] LNetError: 92963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.697803] LNetError: 92963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2520.752185] LNet: Removed LNI 192.168.204.9@tcp [ 2520.842738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2520.968921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2521.026599] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2532.131361] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:28:09 (1785788889) [ 2533.001133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2533.035530] LNet: 93676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.045530] LNetError: 93676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.067119] LNet: Removed LNI 192.168.204.9@tcp [ 2533.119523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2533.193706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2533.223273] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2543.260922] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:28:20 (1785788900) [ 2546.024952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2547.749214] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:28:25 (1785788905) [ 2548.431083] LNet: 94714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.446681] LNetError: 94714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.474338] LNet: Removed LNI 192.168.204.9@tcp [ 2549.098574] Key type .llcrypt unregistered [ 2549.110060] Key type ._llcrypt unregistered [ 2557.376408] Key type ._llcrypt registered [ 2557.378238] Key type .llcrypt registered [ 2557.499704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2557.511309] alg: No test for adler32 (adler32-zlib) [ 2558.510697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2558.563172] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2568.165382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 2575.912994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.109@tcp [ 2576.333430] LNet: 95134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.340931] LNetError: 95134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.358282] LNet: Removed LNI 192.168.204.9@tcp [ 2576.777605] Key type .llcrypt unregistered [ 2576.782095] Key type ._llcrypt unregistered [ 2591.185827] Key type ._llcrypt registered [ 2591.188205] Key type .llcrypt registered [ 2592.941262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2595.126813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2597.231409] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:29:14 (1785788954) [ 2599.256835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2601.113703] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:29:19 (1785788959) [ 2602.995862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg409-server [ 2605.604270] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:29:22 (1785788962) [ 2607.287727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg409-server [ 2609.555303] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:29:26 (1785788966) [ 2612.032376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2612.119026] alg: No test for adler32 (adler32-zlib) [ 2613.388937] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2613.765366] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2615.407794] Key type lgssc registered [ 2617.248939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.020124] Key type lgssc unregistered [ 2646.446440] LNet: 98310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.469454] LNetError: 98310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.494663] LNet: Removed LNI 192.168.204.9@tcp [ 2647.820665] Key type .llcrypt unregistered [ 2647.822154] Key type ._llcrypt unregistered [ 2670.521317] Key type ._llcrypt registered [ 2670.524062] Key type .llcrypt registered [ 2672.135429] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:30:29 (1785789029) [ 2672.776613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2672.786962] alg: No test for adler32 (adler32-zlib) [ 2673.815116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2673.936160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2673.985831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2674.086275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2674.194126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2683.132755] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:30:40 (1785789040) [ 2683.411696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2683.476380] LNet: 99728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2683.533025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2683.631501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2683.680888] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2684.924984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2684.973511] LNet: 99872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.981498] LNetError: 99872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2685.011321] LNet: Removed LNI 192.168.204.9@tcp [ 2685.049696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2685.160227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2685.224317] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2686.208231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2686.257681] LNet: 99900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.260944] LNetError: 99900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2686.278174] LNet: Removed LNI 192.168.204.9@tcp [ 2686.303866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2686.375729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2686.432176] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2694.736672] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:30:52 (1785789052) [ 2694.991427] LNet: 100490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.009979] LNetError: 100490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.046016] LNet: Removed LNI 192.168.204.9@tcp [ 2695.714747] Key type .llcrypt unregistered [ 2695.720981] Key type ._llcrypt unregistered [ 2703.996555] Key type ._llcrypt registered [ 2704.003901] Key type .llcrypt registered [ 2704.387193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2704.418179] alg: No test for adler32 (adler32-zlib) [ 2705.354222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2705.421041] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2705.424076] LNet: *** cfs_fail_loc=e003, val=100*** [ 2706.262488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2706.277921] LNet: *** cfs_fail_loc=e003, val=100*** [ 2706.284449] LNet: Skipped 2 previous similar messages [ 2706.295552] Ping reply received with mapping: [ 2706.301242] found metadata reply NI[192.168.204.9@tcp] [ 2706.306154] 09 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2706.314232] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2706.316676] 5a 5a 5a 5a 5a 5a 5a 5a [ 2706.478166] LNet: 100888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2706.488734] LNetError: 100888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2706.511581] LNet: Removed LNI 192.168.204.9@tcp [ 2707.082687] Key type .llcrypt unregistered [ 2707.085758] Key type ._llcrypt unregistered [ 2724.315193] Key type ._llcrypt registered [ 2724.322660] Key type .llcrypt registered [ 2725.828073] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:31:23 (1785789083) [ 2726.680275] Key type .llcrypt unregistered [ 2726.686337] Key type ._llcrypt unregistered [ 2735.389354] Key type ._llcrypt registered [ 2735.391328] Key type .llcrypt registered [ 2735.543413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2735.552784] alg: No test for adler32 (adler32-zlib) [ 2736.781253] LNetError: 101935:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2737.221631] Key type .llcrypt unregistered [ 2737.223486] Key type ._llcrypt unregistered [ 2743.936210] Key type ._llcrypt registered [ 2743.940727] Key type .llcrypt registered [ 2745.853356] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:31:43 (1785789103) [ 2746.629334] Key type .llcrypt unregistered [ 2746.630689] Key type ._llcrypt unregistered [ 2756.184299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2756.195296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2759.159041] Key type ._llcrypt registered [ 2759.164606] Key type .llcrypt registered [ 2759.460123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2759.494706] alg: No test for adler32 (adler32-zlib) [ 2760.571654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2760.695392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2772.621914] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:32:10 (1785789130) [ 2772.942250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2772.966473] LNet: 103717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.970680] LNetError: 103717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2772.984467] LNet: Removed LNI 10.1.2.3@tcp [ 2773.029398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2774.063575] Key type .llcrypt unregistered [ 2774.065802] Key type ._llcrypt unregistered [ 2782.298335] Key type ._llcrypt registered [ 2782.302707] Key type .llcrypt registered [ 2782.429653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2782.445626] alg: No test for adler32 (adler32-zlib) [ 2783.455691] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2783.524745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2783.752809] LNet: 104005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2783.767086] LNetError: 104005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2783.794278] LNet: Removed LNI 192.168.204.9@tcp [ 2784.571346] Key type .llcrypt unregistered [ 2784.573934] Key type ._llcrypt unregistered [ 2799.926670] Key type ._llcrypt registered [ 2799.930780] Key type .llcrypt registered [ 2801.628307] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:32:38 (1785789158) [ 2802.620316] Key type .llcrypt unregistered [ 2802.624220] Key type ._llcrypt unregistered [ 2810.460298] Key type ._llcrypt registered [ 2810.465312] Key type .llcrypt registered [ 2810.855945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2810.900238] alg: No test for adler32 (adler32-zlib) [ 2819.337674] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:32:57 (1785789177) [ 2819.728717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2819.766721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2819.867401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2819.922413] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2820.883189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2820.929880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2820.965407] LNet: Removed LNI 192.168.204.9@tcp [ 2821.287516] LNet: 105751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2821.906414] Key type .llcrypt unregistered [ 2821.910128] Key type ._llcrypt unregistered [ 2835.450540] Key type ._llcrypt registered [ 2835.452454] Key type .llcrypt registered [ 2836.678390] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:33:14 (1785789194) [ 2837.385137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2837.403292] alg: No test for adler32 (adler32-zlib) [ 2838.446958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2838.543882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2838.588184] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2839.430340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2839.459890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.9@tcp [ 2839.475503] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2839.478932] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2839.482274] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2839, 0, 0 [ 2839.485848] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2839.488392] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2839.491332] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2839.493151] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2839, 0, 0 [ 2839.495646] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2839.501114] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2839.503373] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2839.506707] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2839, 0, 2889 [ 2839.510843] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2839.513608] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2839.516804] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2839.519683] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2839, 0, 2889 [ 2839.526232] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2846.210189] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:33:24 (1785789204) [ 2846.860121] LNet: 107292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2846.869670] LNetError: 107292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2846.891091] LNet: Removed LNI 192.168.204.9@tcp [ 2847.580365] Key type .llcrypt unregistered [ 2847.583956] Key type ._llcrypt unregistered [ 2854.084135] Key type ._llcrypt registered [ 2854.088384] Key type .llcrypt registered [ 2854.457126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2854.486329] alg: No test for adler32 (adler32-zlib) [ 2855.565540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2855.657499] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2865.744951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 2876.679995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2876.759240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2876.805759] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 2878.263752] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2878.528462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2878.532377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2878.540226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2881.151562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2881.192458] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2895.546629] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:34:12 (1785789252) [ 2895.891219] LNet: 108802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2895.908306] LNetError: 108802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2895.917354] LNet: Removed LNI 192.168.204.9@tcp [ 2896.927144] LNet: Removed LNI 192.168.204.9@tcp1 [ 2896.929298] LNetError: 108802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2896.950573] LNetError: 108802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2896.987690] LNet: Removed LNI 10.1.2.3@tcp99 [ 2898.098601] Key type .llcrypt unregistered [ 2898.099898] Key type ._llcrypt unregistered [ 2907.014484] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2907.023585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2909.708292] Key type ._llcrypt registered [ 2909.711700] Key type .llcrypt registered [ 2909.950512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2909.975506] alg: No test for adler32 (adler32-zlib) [ 2910.998687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2911.116068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2911.169298] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2912.162598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2912.196520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2915.398552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2915.479904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2915.534562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2915.566584] LNetError: 109564:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2915.620623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2915.674268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 2916.287579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2916.418700] LNet: 109584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2916.432547] LNetError: 109584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2916.444567] LNet: Removed LNI 10.1.2.3@tcp [ 2917.478747] LNet: Removed LNI 192.168.204.9@tcp [ 2918.421380] Key type .llcrypt unregistered [ 2918.425558] Key type ._llcrypt unregistered [ 2934.277618] Key type ._llcrypt registered [ 2934.281957] Key type .llcrypt registered [ 2936.040331] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:34:53 (1785789293) [ 2936.544881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2936.563400] alg: No test for adler32 (adler32-zlib) [ 2937.447741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2937.488523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2937.505549] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2938.065432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 2938.123959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg409-client@tcp [ 2946.758897] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:35:03 (1785789303) [ 2946.898758] LNet: 111109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2946.917044] LNetError: 111109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2946.947298] LNet: Removed LNI 192.168.204.9@tcp [ 2947.647489] Key type .llcrypt unregistered [ 2947.648673] Key type ._llcrypt unregistered [ 2954.937316] Key type ._llcrypt registered [ 2954.940151] Key type .llcrypt registered [ 2955.160661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2955.182515] alg: No test for adler32 (adler32-zlib) [ 2956.177395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2956.235519] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2956.297514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2966.007878] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:35:23 (1785789323) [ 2968.628417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2970.247966] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:35:28 (1785789328) [ 2972.499903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2974.931183] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:35:32 (1785789332) [ 2975.377667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2975.419684] LNet: 112538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2975.424434] LNetError: 112538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2975.448967] LNet: Removed LNI 192.168.204.9@tcp [ 2975.480534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2975.596990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2975.657938] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 2977.162787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2977.256664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2986.593978] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:35:44 (1785789344) [ 2986.970487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2987.019897] LNet: 113249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2987.041543] LNetError: 113249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2987.073330] LNet: Removed LNI 192.168.204.9@tcp [ 2987.108104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2987.193537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2987.275939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2988.404259] Key type .llcrypt unregistered [ 2988.410737] Key type ._llcrypt unregistered [ 3003.217606] Key type ._llcrypt registered [ 3003.219659] Key type .llcrypt registered [ 3004.335575] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:36:02 (1785789362) [ 3005.378784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3005.419668] alg: No test for adler32 (adler32-zlib) [ 3006.401532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3006.477983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3006.512350] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3007.651930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3007.735555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3015.687396] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:36:13 (1785789373) [ 3016.013373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3016.053986] LNet: 114807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.067136] LNetError: 114807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.081978] LNet: Removed LNI 192.168.204.9@tcp [ 3016.133074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3016.229243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3016.276748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 3024.906739] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:36:22 (1785789382) [ 3025.045331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3025.087668] LNet: 115381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.146416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3025.237455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3025.645240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3025.664518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3025.710985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 3025.999426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3026.038940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3026.133218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3026.155423] LNet: 115421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3026.171127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3026.235587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3026.832756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3026.873792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3026.968625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 3027.564084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3027.616614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3035.413607] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:36:33 (1785789393) [ 3036.007632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3036.028669] LNet: 116022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.062543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3036.139466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3036.205651] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3037.684666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3037.723094] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3038.824268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3038.932324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3039.375870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3039.404821] LNetError: 116302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3039.411544] LNet: Removed LNI 192.168.204.9@tcp [ 3039.435268] LNet: Removed LNI 192.168.204.9@tcp2 [ 3039.485582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3039.610261] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3039.616964] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3039.629260] LNet: 116308:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3040.353329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3040.600225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3040.622201] LNet: 116345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3040.624603] LNet: 116345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3040.636481] LNetError: 116345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3040.643336] LNetError: 116345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3040.650883] LNet: Removed LNI 192.168.204.9@tcp [ 3040.673910] LNet: Removed LNI 192.168.204.9@tcp2 [ 3040.723508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3040.801419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3040.828182] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3041.665659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3041.701923] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3043.101241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3043.167425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3043.279178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3043.730705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3043.781155] LNetError: 116630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3043.794403] LNetError: 116630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3043.813348] LNet: Removed LNI 192.168.204.9@tcp [ 3044.851944] LNet: Removed LNI 192.168.204.9@tcp2 [ 3044.914963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3045.090659] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3045.105884] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3045.124450] LNet: 116636:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3045.961639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3046.224518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3046.257388] LNet: Removed LNI 192.168.204.9@tcp [ 3047.335333] LNet: Removed LNI 192.168.204.9@tcp2 [ 3047.414341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3047.495153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3047.543661] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3048.772086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3048.797753] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3050.475919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3050.608344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3051.033072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3051.086226] LNet: 116953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3051.096119] LNet: 116953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3051.124900] LNetError: 116953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3051.135730] LNetError: 116953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 3052.191186] LNet: Removed LNI 192.168.204.9@tcp [ 3052.206844] LNet: Removed LNI 192.168.204.9@tcp2 [ 3052.326668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3052.478062] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3052.497961] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3052.507451] LNet: 116959:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3053.645847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3054.088929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3055.135125] LNet: Removed LNI 192.168.204.9@tcp [ 3055.173969] LNet: Removed LNI 192.168.204.9@tcp2 [ 3055.239572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3055.352422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3055.399928] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3057.110535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3057.175646] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3058.485601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3058.560080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3058.641355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3058.949889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3060.002220] LNet: Removed LNI 192.168.204.9@tcp [ 3060.026149] LNetError: 117282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3060.038968] LNetError: 117282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3060.065298] LNet: Removed LNI 192.168.204.9@tcp2 [ 3060.160449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3060.280312] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3060.293134] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3060.300420] LNet: 117288:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3061.557297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3061.994619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3062.035937] LNet: Removed LNI 192.168.204.9@tcp [ 3063.078832] LNet: Removed LNI 192.168.204.9@tcp2 [ 3063.157810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3063.299864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3063.388448] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3065.059698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3065.142093] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3066.718194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3066.768727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3066.866082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3067.499361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3067.513684] LNet: 117622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3067.522325] LNet: 117622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3068.575232] LNet: Removed LNI 192.168.204.9@tcp [ 3068.590615] LNet: Removed LNI 192.168.204.9@tcp2 [ 3068.658784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3068.793246] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3068.802952] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3068.813949] LNet: 117629:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3069.664786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3070.232179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3071.327156] LNet: Removed LNI 192.168.204.9@tcp [ 3071.347755] LNet: Removed LNI 192.168.204.9@tcp2 [ 3071.435187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3071.597617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3071.657623] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3072.912689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3072.971083] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3074.491666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3074.585814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3074.658657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3074.767609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3075.422343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3076.513630] LNet: Removed LNI 192.168.204.9@tcp [ 3076.558972] LNetError: 117983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3076.571576] LNetError: 117983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3076.604450] LNet: Removed LNI 192.168.204.9@tcp2 [ 3076.742314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3076.874695] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3076.886852] LNet: Added LNI 192.168.204.9@tcp2 [8/256/0/180] [ 3076.903863] LNet: 117989:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3077.845964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3086.881165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3089.382979] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:37:26 (1785789446) [ 3089.737192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3089.803863] LNet: Removed LNI 192.168.204.9@tcp [ 3090.855407] LNet: Removed LNI 192.168.204.9@tcp2 [ 3090.922057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3091.358644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3091.409453] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3092.522379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3092.545560] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3094.031249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3094.079250] LNet: Removed LNI 10.1.2.3@tcp [ 3094.114607] LNet: Removed LNI 192.168.204.9@tcp [ 3094.168773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3094.317493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3094.375687] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3094.385574] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3095.217562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3095.258594] LNet: Removed LNI 192.168.204.9@tcp [ 3095.283256] LNet: Removed LNI 10.1.2.3@tcp [ 3104.014494] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 16:37:41 (1785789461) [ 3104.388581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3104.409277] LNet: 119625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3104.415590] LNet: 119625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 3104.452719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3104.521982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3104.551779] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3105.545121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3105.605247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3105.680556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3105.778671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3105.838482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3105.913307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3105.953491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3106.024660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3106.088465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3106.134024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3106.193967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.9@tcp [ 3106.451309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3115.221844] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:37:52 (1785789472) [ 3115.696078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3115.767644] LNetError: 120365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3115.775584] LNetError: 120365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3115.815805] LNet: Removed LNI 192.168.204.9@tcp [ 3115.890185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3116.058996] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3116.166143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3116.177995] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3116.190845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3117.723942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3117.817654] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3119.066457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3119.106123] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3123.128730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.9@tcp [ 3123.214904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3123.313689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.9@tcp -m PUT -e local_timeout -r 1 [ 3123.409050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3123.484894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3123.491118] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3123.889571] LNetError: 120424:0:(lib-move.c:4801:lnet_parse()) 192.168.204.9@tcp, src 192.168.204.9@tcp: Dropping PUT (error -108 looking up sender) [ 3125.920259] LNet: Removed LNI 10.1.2.3@tcp [ 3127.015208] LNet: Removed LNI 192.168.204.9@tcp [ 3127.860865] Key type .llcrypt unregistered [ 3127.867463] Key type ._llcrypt unregistered [ 3145.506473] Key type ._llcrypt registered [ 3145.508602] Key type .llcrypt registered [ 3146.792424] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:38:24 (1785789504) [ 3147.508248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3147.533238] alg: No test for adler32 (adler32-zlib) [ 3148.570182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3149.048144] LNet: 121738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.112325] Key type .llcrypt unregistered [ 3150.113854] Key type ._llcrypt unregistered [ 3157.948098] Key type ._llcrypt registered [ 3157.953303] Key type .llcrypt registered [ 3158.298953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3158.337387] alg: No test for adler32 (adler32-zlib) [ 3159.430894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3159.488459] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3170.166579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 3170.456606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3171.230657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.109@tcp [ 3171.471500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3171.525803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3171.584990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3171.634846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3171.690383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3171.736616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3171.794724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3171.822931] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 3171.877916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3171.914684] LNet: 122190:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3171.947708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3171.987321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3172.029206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3172.081652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3172.127490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3172.172107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3173.038303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3173.301179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3173.337608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3173.388277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3173.422350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3173.464255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3173.524660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3173.585292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3173.629531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3173.667816] LNet: 122307:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3173.695640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3173.746656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3173.792457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3173.845629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3173.882568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3173.931017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3174.769802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.109@tcp [ 3175.099332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3175.132482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3175.180700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3175.234944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3175.272830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3175.343066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3175.384860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3175.433907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3175.457043] LNet: 122425:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3175.490522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3175.535677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3175.575913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3175.604943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3175.643921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3175.684827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3176.290116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3176.578374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3176.623811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3176.679511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3176.716866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3176.758942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3176.804293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3176.856392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3176.910331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3176.945111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3176.983468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3177.025556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3177.059619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3177.105574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3177.159846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3177.706418] LNet: 122588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.714963] LNetError: 122588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.739759] LNet: Removed LNI 192.168.204.9@tcp [ 3178.365710] Key type .llcrypt unregistered [ 3178.369871] Key type ._llcrypt unregistered [ 3196.434674] Key type ._llcrypt registered [ 3196.439256] Key type .llcrypt registered [ 3198.019756] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:39:15 (1785789555) [ 3198.666748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3198.690797] alg: No test for adler32 (adler32-zlib) [ 3199.748618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3200.298819] LNet: 123485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3201.058779] Key type .llcrypt unregistered [ 3201.062111] Key type ._llcrypt unregistered [ 3207.829851] Key type ._llcrypt registered [ 3207.834579] Key type .llcrypt registered [ 3208.295969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3208.340569] alg: No test for adler32 (adler32-zlib) [ 3209.405311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3209.472714] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3218.825082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.109@tcp [ 3228.964399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3229.040936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3229.071781] LNet: Added LNI 192.168.204.9@tcp1 [8/256/0/180] [ 3230.538583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3231.368557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.109@tcp [ 3231.784798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3231.846124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3231.895481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3231.958612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3232.008559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3232.033133] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.109@tcp [ 3232.057848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3232.109615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3232.143830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3232.167601] LNet: 124140:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3232.202103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3232.250432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3232.293677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3232.333657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3232.373918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3232.428406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3233.125374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3233.378937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3233.445622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3233.498867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3233.543821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3233.591467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3233.621913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3233.647677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3233.680633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3233.706398] LNet: 124257:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3233.726591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3233.755526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3233.784547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3233.807841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3233.840583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3233.872685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3234.471830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.109@tcp [ 3234.751691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3234.799228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3234.829471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3234.856453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3234.886386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3234.907912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3234.931885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3234.966662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3234.981331] LNet: 124374:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3234.998874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3235.019630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3235.047627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3235.081280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3235.111949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3235.138702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3235.628664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3235.813411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3235.836766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3235.861880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3235.887820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3235.923336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3235.958607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3235.985327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3236.012420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3236.039477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3236.073672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3236.101392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3236.135526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3236.166753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3236.191542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3242.137624] LNet: 124573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.154503] LNetError: 124573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.161643] LNet: Removed LNI 192.168.204.9@tcp [ 3242.181172] LNet: Removed LNI 192.168.204.9@tcp1 [ 3242.810803] Key type .llcrypt unregistered [ 3242.814554] Key type ._llcrypt unregistered [ 3264.682392] Key type ._llcrypt registered [ 3264.691329] Key type .llcrypt registered [ 3267.631300] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:40:23 (1785789623) [ 3270.770403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3272.837981] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:40:30 (1785789630) [ 3284.449613] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:40:41 (1785789641) [ 3296.169418] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:40:53 (1785789653) [ 3307.788383] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:41:04 (1785789664) [ 3318.038075] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:41:15 (1785789675) [ 3328.944379] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:41:26 (1785789686) [ 3339.244450] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:41:36 (1785789696) [ 3348.091851] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:41:45 (1785789705) [ 3356.311325] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:41:53 (1785789713) [ 3365.203471] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:42:02 (1785789722) [ 3373.611157] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:42:11 (1785789731) [ 3383.530616] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:42:21 (1785789741) [ 3393.455110] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:42:31 (1785789751) [ 3403.649725] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:42:40 (1785789760) [ 3412.743479] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:42:50 (1785789770) [ 3421.406947] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:42:59 (1785789779) [ 3429.316633] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:43:07 (1785789787) [ 3437.550961] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:43:15 (1785789795) [ 3438.341620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.351130] alg: No test for adler32 (adler32-zlib) [ 3439.375606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3439.572253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3439.603338] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3441.195377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3441.230175] LNet: 135585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.239904] LNetError: 135585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.266765] LNet: Removed LNI 192.168.204.9@tcp [ 3441.290574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3441.372853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3441.409541] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3450.318806] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:43:28 (1785789808) [ 3458.856973] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:43:36 (1785789816) [ 3469.438169] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:43:46 (1785789826) [ 3480.326707] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:43:58 (1785789838) [ 3490.510278] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:44:07 (1785789847) [ 3499.813639] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:44:17 (1785789857) [ 3509.410175] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:44:26 (1785789866) [ 3518.357199] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:44:35 (1785789875) [ 3526.461556] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:44:44 (1785789884) [ 3534.345810] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:44:52 (1785789892) [ 3543.506986] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:45:01 (1785789901) [ 3552.301209] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:45:09 (1785789909) [ 3561.097532] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3208 sec ========== 16:45:18 (1785789918) [ 3561.563492] LNet: 143057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.571074] LNetError: 143057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.584839] LNet: Removed LNI 192.168.204.9@tcp [ 3562.419712] Key type .llcrypt unregistered [ 3562.424570] Key type ._llcrypt unregistered [ 3579.634077] Key type ._llcrypt registered [ 3579.640616] Key type .llcrypt registered [ 3579.946174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3579.966503] alg: No test for adler32 (adler32-zlib) [ 3581.017561] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3581.286991] LNet: Added LNI 192.168.204.9@tcp [8/256/0/180] [ 3583.079625] Key type lgssc registered [ 3584.805743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3659.184668] Lustre: Mounted lustre-client [ 3665.113238] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3675.897677] Lustre: Unmounted lustre-client [ 3698.984031] Key type lgssc unregistered [ 3699.312099] LNet: 145988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3699.319955] LNetError: 145988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3699.355765] LNet: Removed LNI 192.168.204.9@tcp [ 3700.093150] Key type .llcrypt unregistered [ 3700.095488] Key type ._llcrypt unregistered