[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443789248 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003227] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008416] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014087] *** VALIDATE selinux *** [ 0.021992] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025309] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029093] *** VALIDATE tmpfs *** [ 0.031044] *** VALIDATE proc *** [ 0.032235] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037028] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041381] debug: unmapping init [mem 0xffffffff9de59000-0xffffffff9de60fff] [ 0.042482] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043660] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044050] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051264] rcu: Hierarchical SRCU implementation. [ 0.053376] smp: Bringing up secondary CPUs ... [ 0.054545] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.058067] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144015] node 0 deferred pages initialised in 81ms [ 0.147341] devtmpfs: initialized [ 0.148205] x86/mm: Memory block size: 128MB [ 0.150777] gcov: version magic: 0x41383552 [ 0.151616] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155282] pinctrl core: initialized pinctrl subsystem [ 0.157137] [ 0.157514] ************************************************************* [ 0.159010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161010] ** ** [ 0.163010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165011] ** ** [ 0.167068] ** This means that this kernel is built to expose internal ** [ 0.169011] ** IOMMU data structures, which may compromise security on ** [ 0.171010] ** your system. ** [ 0.173012] ** ** [ 0.175009] ** If you see this message and you are not debugging the ** [ 0.177010] ** kernel, report this immediately to your vendor! ** [ 0.179009] ** ** [ 0.181012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183010] ************************************************************* [ 0.185604] NET: Registered protocol family 16 [ 0.187418] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195591] cpuidle: using governor menu [ 0.196817] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198484] PCI: Using configuration type 1 for base access [ 0.199181] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213066] cryptd: max_cpu_qlen set to 1000 [ 0.216230] ACPI: Added _OSI(Module Device) [ 0.217014] ACPI: Added _OSI(Processor Device) [ 0.218011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222863] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228501] ACPI: Interpreter enabled [ 0.230062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.232014] ACPI: Using IOAPIC for interrupt routing [ 0.234113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.247563] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.257077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.262258] acpiphp: Slot [2] registered [ 0.263118] acpiphp: Slot [5] registered [ 0.265108] acpiphp: Slot [6] registered [ 0.267083] acpiphp: Slot [3] registered [ 0.268090] acpiphp: Slot [4] registered [ 0.270074] acpiphp: Slot [7] registered [ 0.271068] acpiphp: Slot [8] registered [ 0.272092] acpiphp: Slot [9] registered [ 0.273076] acpiphp: Slot [10] registered [ 0.274093] acpiphp: Slot [11] registered [ 0.276130] acpiphp: Slot [12] registered [ 0.279105] acpiphp: Slot [13] registered [ 0.281115] acpiphp: Slot [14] registered [ 0.283113] acpiphp: Slot [15] registered [ 0.285142] acpiphp: Slot [16] registered [ 0.287129] acpiphp: Slot [17] registered [ 0.290170] acpiphp: Slot [18] registered [ 0.291141] acpiphp: Slot [19] registered [ 0.293207] acpiphp: Slot [20] registered [ 0.295133] acpiphp: Slot [21] registered [ 0.296181] acpiphp: Slot [22] registered [ 0.298159] acpiphp: Slot [23] registered [ 0.299163] acpiphp: Slot [24] registered [ 0.300141] acpiphp: Slot [25] registered [ 0.302147] acpiphp: Slot [26] registered [ 0.304165] acpiphp: Slot [27] registered [ 0.305216] acpiphp: Slot [28] registered [ 0.307135] acpiphp: Slot [29] registered [ 0.309105] acpiphp: Slot [30] registered [ 0.311131] acpiphp: Slot [31] registered [ 0.313090] PCI host bridge to bus 0000:00 [ 0.315031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.331037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339309] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343399] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.347661] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.360065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372799] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384599] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.401021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412067] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.461708] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.468025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.476021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522113] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.524508] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.527426] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.530423] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.533286] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.539260] iommu: Default domain type: Passthrough [ 0.542622] SCSI subsystem initialized [ 0.544156] ACPI: bus type USB registered [ 0.546272] usbcore: registered new interface driver usbfs [ 0.548156] usbcore: registered new interface driver hub [ 0.550164] usbcore: registered new device driver usb [ 0.552240] pps_core: LinuxPPS API ver. 1 registered [ 0.555020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.558085] PTP clock support registered [ 0.560373] EDAC MC: Ver: 3.0.0 [ 0.562224] PCI: Using ACPI for IRQ routing [ 0.563522] NetLabel: Initializing [ 0.564021] NetLabel: domain hash size = 128 [ 0.566011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.567077] NetLabel: unlabeled traffic allowed by default [ 0.570138] vgaarb: loaded [ 0.571302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.574028] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.579350] clocksource: Switched to clocksource kvm-clock [ 0.683118] VFS: Disk quotas dquot_6.6.0 [ 0.684251] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.685766] *** VALIDATE ramfs *** [ 0.686673] *** VALIDATE hugetlbfs *** [ 0.687812] pnp: PnP ACPI init [ 0.689677] pnp: PnP ACPI: found 6 devices [ 0.708562] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.710987] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.712518] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.713945] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.715578] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.717400] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.719315] NET: Registered protocol family 2 [ 0.721155] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.728313] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.732191] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.737833] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.740877] TCP: Hash tables configured (established 65536 bind 65536) [ 0.743753] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.746554] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.749520] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.752590] NET: Registered protocol family 1 [ 0.755099] RPC: Registered named UNIX socket transport module. [ 0.756921] RPC: Registered udp transport module. [ 0.758745] RPC: Registered tcp transport module. [ 0.760688] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.763475] NET: Registered protocol family 44 [ 0.765512] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.767894] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.770496] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.773868] PCI: CLS 0 bytes, default 64 [ 0.776131] Unpacking initramfs... [ 2.256278] debug: unmapping init [mem 0xffff98fe7cc64000-0xffff98fe7ffcffff] [ 2.262802] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.265427] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.269992] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.768493] Initialise system trusted keyrings [ 2.770551] Key type blacklist registered [ 2.774117] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.786430] zbud: loaded [ 2.789758] *** VALIDATE nfs *** [ 2.791076] *** VALIDATE nfs4 *** [ 2.792712] pstore: using deflate compression [ 2.795639] Platform Keyring initialized [ 2.918286] NET: Registered protocol family 38 [ 2.919753] Key type asymmetric registered [ 2.921083] Asymmetric key parser 'x509' registered [ 2.923158] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.925979] io scheduler mq-deadline registered [ 2.927529] io scheduler kyber registered [ 2.929338] io scheduler bfq registered [ 2.931290] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.934619] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.937558] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.940556] ACPI: Power Button [PWRF] [ 2.947257] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.954525] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.967678] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.995847] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.024730] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.030185] Non-volatile memory driver v1.3 [ 3.031368] Linux agpgart interface v0.103 [ 3.069252] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.073511] vda: detected capacity change from 0 to 74702848 [ 3.088867] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.091696] vdb: detected capacity change from 0 to 1073741824 [ 3.108696] libphy: Fixed MDIO Bus: probed [ 3.115511] usbcore: registered new interface driver usbserial_generic [ 3.118654] usbserial: USB Serial support registered for generic [ 3.121745] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.127451] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.129621] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.132795] mousedev: PS/2 mouse device common for all mice [ 3.136532] rtc_cmos 00:05: RTC can wake from S4 [ 3.139803] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.140246] rtc_cmos 00:05: registered as rtc0 [ 3.145748] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.146889] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.149729] intel_pstate: CPU model not supported [ 3.155883] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.160021] hid: raw HID events driver (C) Jiri Kosina [ 3.162505] usbcore: registered new interface driver usbhid [ 3.164721] usbhid: USB HID core driver [ 3.166428] drop_monitor: Initializing network drop monitor service [ 3.168996] Initializing XFRM netlink socket [ 3.171189] NET: Registered protocol family 10 [ 3.174966] Segment Routing with IPv6 [ 3.176527] NET: Registered protocol family 17 [ 3.179242] mpls_gso: MPLS GSO support [ 3.185802] RAS: Correctable Errors collector initialized. [ 3.188248] AVX version of gcm_enc/dec engaged. [ 3.190209] AES CTR mode by8 optimization enabled [ 3.273300] sched_clock: Marking stable (3273276597, 0)->(4161439899, -888163302) [ 3.277157] registered taskstats version 1 [ 3.280115] Loading compiled-in X.509 certificates [ 3.282241] zswap: loaded using pool lzo/zbud [ 3.310334] Key type big_key registered [ 3.322951] Key type encrypted registered [ 3.324811] ima: No TPM chip found, activating TPM-bypass! [ 3.327344] ima: Allocated hash algorithm: sha1 [ 3.329332] ima: No architecture policies found [ 3.331255] evm: Initialising EVM extended attributes: [ 3.333348] evm: security.selinux [ 3.334702] evm: security.ima [ 3.335972] evm: security.capability [ 3.337481] evm: HMAC attrs: 0x1 [ 3.339867] rtc_cmos 00:05: setting system clock to 2026-08-15 01:04:22 UTC (1786755862) [ 3.346602] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.349966] debug: unmapping init [mem 0xffffffff9db82000-0xffffffff9de58fff] [ 3.360155] Write protecting the kernel read-only data: 28672k [ 3.364032] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 3.367080] debug: unmapping init [mem 0xffffffff9cb14000-0xffffffff9cbfffff] [ 3.399136] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.408634] systemd[1]: Detected virtualization kvm. [ 3.410707] systemd[1]: Detected architecture x86-64. [ 3.412790] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.440496] systemd[1]: No hostname configured. [ 3.442478] systemd[1]: Set hostname to . [ 3.444843] random: systemd: uninitialized urandom read (16 bytes read) [ 3.447693] systemd[1]: Initializing machine ID from random generator. [ 3.491360] random: ln: uninitialized urandom read (6 bytes read) [ 3.576879] random: systemd: uninitialized urandom read (16 bytes read) [ 3.579737] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.583665] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.588096] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.323518] device-mapper: uevent: version 1.0.3 [ 4.327242] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.194138] virtio_net virtio0 ens2: renamed from eth0 [ 6.272996] random: fast init done [ 6.461437] scsi host0: ata_piix [ 6.631192] scsi host1: ata_piix [ 6.634683] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.643861] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.852884] random: crng init done [ 11.855428] random: 7 urandom warning(s) missed due to ratelimiting [ 12.535718] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.019680] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.209620] printk: systemd: 24 output lines suppressed due to ratelimiting [ 19.160534] SELinux: Disabled at runtime. [ 19.326664] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.356692] systemd[1]: Detected virtualization kvm. [ 19.360398] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.019088] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.026987] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.061696] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.071041] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.077541] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.105382] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.125574] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ 21.557042] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.610425] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.630066] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.712174] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.387407] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.598142] EDAC sbridge: Ver: 1.1.2 [ 27.523941] Key type dns_resolver registered [ 28.201321] NFS: Registering the id_resolver key type [ 28.204267] Key type id_resolver registered [ 28.206415] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg607-client login: [ 39.816250] hrtimer: interrupt took 6781690 ns [ 86.109820] libcfs: loading out-of-tree module taints kernel. [ 86.169626] Key type ._llcrypt registered [ 86.175239] Key type .llcrypt registered [ 86.639200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 86.649131] alg: No test for adler32 (adler32-zlib) [ 88.210930] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 89.077318] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 90.792199] Key type lgssc registered [ 92.405034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 207.301317] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6 [ 211.753966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.567896] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing check_logdir /tmp/testlogs/ [ 227.026991] Lustre: DEBUG MARKER: oleg607-client.virtnet: executing yml_node [ 231.055936] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 232.928147] Lustre: lustre-OST0000-osc-ffff98fec9853000: disconnect after 24s idle [ 233.590914] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 236.354479] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 237.861708] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 21:08:15 EDT 2026 [ 251.888843] Lustre: Unmounted lustre-client [ 270.427024] Key type lgssc unregistered [ 270.762191] LNet: 8404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.770881] LNetError: 8404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 270.787521] LNet: Removed LNI 192.168.206.7@tcp [ 271.663230] Key type .llcrypt unregistered [ 271.665990] Key type ._llcrypt unregistered [ 300.870800] Key type ._llcrypt registered [ 300.874820] Key type .llcrypt registered [ 301.057148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 301.073740] alg: No test for adler32 (adler32-zlib) [ 301.966756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 302.052857] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 302.093162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 306.130907] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 306.266605] LNet: 9286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.282108] LNetError: 9286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.311826] LNet: Removed LNI 192.168.206.7@tcp [ 306.985461] Key type .llcrypt unregistered [ 306.988575] Key type ._llcrypt unregistered [ 312.505951] Key type ._llcrypt registered [ 312.511373] Key type .llcrypt registered [ 313.769766] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:09:31 (1786756171) [ 314.161209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 314.184405] alg: No test for adler32 (adler32-zlib) [ 315.078695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.141484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.165361] LNet: 9709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.190374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.227759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 315.246281] LNet: 9716:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 321.195896] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:09:39 (1786756179) [ 321.272180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.305216] LNet: 10287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.330441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.427982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 326.231824] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:09:44 (1786756184) [ 326.537940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.563506] LNet: 10877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.576916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.622801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 332.848673] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:09:50 (1786756190) [ 332.926734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.961771] LNet: 11454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.990805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.087701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 339.132277] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:09:57 (1786756197) [ 339.469612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.490804] LNet: 12047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.513963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.588723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 340.147327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 346.082969] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:10:04 (1786756204) [ 346.176986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.201521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.246863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 346.655143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 346.963968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 352.150564] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:10:10 (1786756210) [ 352.244148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.265668] LNet: 13228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.269906] LNet: 13228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 352.289077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.413816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 358.587969] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:10:16 (1786756216) [ 358.702260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.770379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.859573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 358.913458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.356247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 359.418884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.911122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 359.954326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.367648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 360.392321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.770367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 360.835664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.318604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 361.383721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.857695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 361.906958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.422525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 362.459447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.918964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 362.959400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.399223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 363.446431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 363.979227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 364.031100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.329384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 364.360381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 364.760453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 364.805159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 370.537799] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:10:28 (1786756228) [ 370.695340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.733773] LNet: 14501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.736559] LNet: 14501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 370.761152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.809701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 370.844141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 377.152451] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:10:35 (1786756235) [ 377.309321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.363182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.444690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 377.502248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 383.932805] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:10:41 (1786756241) [ 384.065356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.113654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.224441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 384.255552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 390.837930] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:10:48 (1786756248) [ 391.101744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.145793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.235378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 391.285025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 396.871810] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:10:54 (1786756254) [ 397.143893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.185760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.294661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 397.326764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 403.040762] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:11:00 (1786756260) [ 403.121095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.172533] LNet: 17426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.178064] LNet: 17426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 403.212740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.319533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 403.360943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 408.834897] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:11:06 (1786756266) [ 408.927940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.957493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.025883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.482967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 415.069413] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:11:12 (1786756272) [ 415.198721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.254865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.451332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 415.654806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.378874] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:11:19 (1786756279) [ 421.519414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.580524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.676406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 421.848730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.980970] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:11:25 (1786756285) [ 428.084307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.147278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.642856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.929568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.712819] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:11:32 (1786756292) [ 434.795894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.868620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.982290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.034684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.607941] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:11:38 (1786756298) [ 440.691662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.749250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.849072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.885411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.429390] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:11:44 (1786756304) [ 446.524223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.577103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.698324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 451.811599] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:11:49 (1786756309) [ 452.047801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.080281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.142146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 457.222595] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:11:55 (1786756315) [ 457.560875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.605343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.653483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 458.071269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 463.233770] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:12:01 (1786756321) [ 463.385559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.417951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.468858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 463.508073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 468.761761] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:12:06 (1786756326) [ 469.139178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.168520] LNet: 24045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.175127] LNet: 24045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 469.200348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.282284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 469.341094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 474.536590] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:12:12 (1786756332) [ 474.639873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.680806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.742252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 474.765976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 480.319863] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:12:18 (1786756338) [ 480.466366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.511370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.597688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 480.642770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 487.014266] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:12:25 (1786756345) [ 487.255412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.308691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.357743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 487.429220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 487.449715] LNetError: 25816:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 487.488503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 487.519851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 492.873866] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:12:30 (1786756350) [ 493.061725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.118669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.228935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oNOPw1z748 [ 493.292849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oNOPw1z748 [ 493.351539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.oNOPw1z748 [ 499.045806] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:12:36 (1786756356) [ 499.176969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.245896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.320321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 499.351941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 499.405292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 505.165445] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:12:42 (1786756362) [ 505.325553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.371805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.447306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 505.478348] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 506.763615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 507.947633] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 509.848211] Key type lgssc registered [ 510.995321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 514.474566] LNetError: 27577:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.7@tcp99 has no available nets [ 517.257972] Key type lgssc unregistered [ 517.559184] LNetError: 28686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.572803] LNet: Removed LNI 192.168.206.7@tcp [ 518.162591] Key type .llcrypt unregistered [ 518.167393] Key type ._llcrypt unregistered [ 524.141327] Key type ._llcrypt registered [ 524.145370] Key type .llcrypt registered [ 525.555060] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:13:03 (1786756383) [ 526.051730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 526.064162] alg: No test for adler32 (adler32-zlib) [ 527.035785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.135202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 530.240173] LNet: Message forwarding will be enabled in 2s [ 533.344133] LNet: Message forwarding enabled [ 533.855844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 533.882471] LNet: Message forwarding will stop in 2s to 6s [ 538.239919] LNet: Message forwarding disabled [ 538.486228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 541.005894] LNet: Message forwarding will be enabled in 2s [ 544.425431] LNet: Message forwarding enabled [ 547.820514] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:13:25 (1786756405) [ 547.945245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.002367] LNet: 30088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.175823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.272664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 554.700523] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:13:32 (1786756412) [ 554.950775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.986070] LNet: 30675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.017789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.101596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 555.144647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 555.191699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 555.218493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 555.255319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 555.292283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 555.321247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 555.363276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 555.412121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 555.469801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 555.522892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 555.555618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 555.597157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 555.630852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 555.679651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 555.730384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 555.790403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 555.870393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 555.965682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 556.004253] LNetError: 30718:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 561.560801] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:13:39 (1786756419) [ 561.643968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.663991] LNet: 31283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.683461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.767833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.570744] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:13:47 (1786756427) [ 571.607573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg607-server [ 573.566802] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:13:51 (1786756431) [ 573.680484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.722212] LNet: 32215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.761258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.873399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.903241] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 574.855677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[8-12]@tcp [ 574.877930] LNet: 32353:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 574.964833] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.8@tcp is being used as a gateway but routing feature is not turned on [ 575.004836] LNetError: 32223:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.8@tcp has gone from up to down [ 575.986114] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.8@tcp is being used as a gateway but routing feature is not turned on [ 576.003766] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 576.996885] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 577.022029] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 579.043745] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.8@tcp is being used as a gateway but routing feature is not turned on [ 579.051428] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 581.849501] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:13:59 (1786756439) [ 583.139597] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.9@tcp is being used as a gateway but routing feature is not turned on [ 583.153484] LNetError: 32223:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 583.858105] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg607-server [ 585.348266] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:14:03 (1786756443) [ 585.558984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.582553] LNet: 33288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.588973] LNetError: 33288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.603492] LNet: Removed LNI 192.168.206.7@tcp [ 585.624844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.684800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.721407] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 586.441559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[8-13/2]@tcp [ 586.467950] LNet: 33406:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 586.473995] LNet: 33406:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 586.493412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[8-13/2]@tcp [ 586.498192] LNetError: 33292:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.10@tcp has gone from up to down [ 586.530945] LNetError: 33292:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 592.073613] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:14:09 (1786756449) [ 592.132711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.177599] LNetError: 33993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.187451] LNet: Removed LNI 192.168.206.7@tcp [ 592.210372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.317960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 592.372701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.408333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.457881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.508499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.556547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.623551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 592.709654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.737917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.806414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.882362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.911518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.995892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 593.100436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.145542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.223656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.348497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.385617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.448619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 593.544717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.591111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.653685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.738891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.779519] LNet: 34105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.785563] LNet: 34105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 593.805206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.882261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 593.958588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.013892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.061523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.175177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.238918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.291423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 594.378103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.410213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.504282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.777915] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:14:19 (1786756459) [ 604.413950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg607-server [ 606.570346] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:14:24 (1786756464) [ 608.918977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg607-server [ 610.782927] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:14:28 (1786756468) [ 610.958423] LNet: 35392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.972502] LNet: 35392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 611.772469] Key type .llcrypt unregistered [ 611.775831] Key type ._llcrypt unregistered [ 617.435275] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 617.652549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 617.663315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 617.674092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 620.102578] Key type ._llcrypt registered [ 620.105813] Key type .llcrypt registered [ 620.248339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.259698] alg: No test for adler32 (adler32-zlib) [ 621.184192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.269946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.315396] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 622.111969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 622.131688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 623.740444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 623.770304] LNet: Removed LNI 10.1.2.3@tcp [ 623.890961] LNet: 36124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.903266] LNetError: 36124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.919834] LNet: Removed LNI 192.168.206.7@tcp [ 624.480664] Key type .llcrypt unregistered [ 624.483399] Key type ._llcrypt unregistered [ 634.408788] Key type ._llcrypt registered [ 634.411602] Key type .llcrypt registered [ 635.620573] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:14:53 (1786756493) [ 636.131264] Key type .llcrypt unregistered [ 636.133944] Key type ._llcrypt unregistered [ 641.193602] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 641.348023] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 641.357869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 641.363422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 643.833697] Key type ._llcrypt registered [ 643.835514] Key type .llcrypt registered [ 643.997969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 644.018296] alg: No test for adler32 (adler32-zlib) [ 644.911659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.961092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.987518] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 645.532843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 645.552258] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.556574] LNet: 37619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.576599] LNetError: 37619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 647.586060] LNet: Removed LNI 10.1.2.3@tcp [ 647.597640] LNet: Removed LNI 192.168.206.7@tcp [ 648.174161] Key type .llcrypt unregistered [ 648.178791] Key type ._llcrypt unregistered [ 658.951739] Key type ._llcrypt registered [ 658.955822] Key type .llcrypt registered [ 660.134962] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:15:17 (1786756517) [ 660.592589] Key type .llcrypt unregistered [ 660.595840] Key type ._llcrypt unregistered [ 666.187859] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.356443] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.370393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.374378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 668.833500] Key type ._llcrypt registered [ 668.835826] Key type .llcrypt registered [ 669.013152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.035346] alg: No test for adler32 (adler32-zlib) [ 669.981500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.033218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 670.049609] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 671.697978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 671.720637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 673.457989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 673.488164] LNet: Removed LNI 10.1.2.3@tcp [ 673.505580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 673.528152] LNet: Removed LNI 10.1.2.31@tcp [ 673.551965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.588137] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.341773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 674.362292] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 675.160669] LNet: 39545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.167072] LNetError: 39545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 675.173546] LNet: Removed LNI 10.1.2.31@tcp [ 675.184880] LNet: Removed LNI 10.1.2.3@tcp [ 675.717547] Key type .llcrypt unregistered [ 675.719375] Key type ._llcrypt unregistered [ 685.982921] Key type ._llcrypt registered [ 685.984804] Key type .llcrypt registered [ 687.204937] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:15:45 (1786756545) [ 687.633757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.653849] alg: No test for adler32 (adler32-zlib) [ 688.612520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.668925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.695243] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 689.853295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.881959] LNet: 40502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.910995] LNetError: 40502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.928398] LNet: Removed LNI 192.168.206.7@tcp [ 689.966249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.033441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.7@tcp [ 690.065996] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 690.856676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.7@tcp [ 690.888959] LNet: Removed LNI 192.168.206.7@tcp [ 690.966593] LNet: 40637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.809775] Key type .llcrypt unregistered [ 691.816612] Key type ._llcrypt unregistered [ 702.767734] Key type ._llcrypt registered [ 702.769781] Key type .llcrypt registered [ 703.910110] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:16:01 (1786756561) [ 705.464490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg607-server [ 707.076352] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:16:04 (1786756564) [ 707.863235] Key type .llcrypt unregistered [ 707.871318] Key type ._llcrypt unregistered [ 714.801513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 715.027120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 715.052590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 715.068473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 717.575218] Key type ._llcrypt registered [ 717.577174] Key type .llcrypt registered [ 717.765326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.793607] alg: No test for adler32 (adler32-zlib) [ 718.693847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.782829] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 719.674759] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 721.617557] LNet: 42476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.628279] LNetError: 42476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.633674] LNet: Removed LNI 192.168.206.7@tcp1 [ 721.644590] LNet: Removed LNI 10.1.2.3@tcp2 [ 722.049923] Key type .llcrypt unregistered [ 722.051601] Key type ._llcrypt unregistered [ 732.536423] Key type ._llcrypt registered [ 732.538644] Key type .llcrypt registered [ 733.767900] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:16:31 (1786756591) [ 734.364938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.384784] alg: No test for adler32 (adler32-zlib) [ 735.388511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.453373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.475486] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 736.193271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.211344] LNet: 43449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.217276] LNetError: 43449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.227691] LNet: Removed LNI 192.168.206.7@tcp [ 736.244897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.298344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 736.322751] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 742.772905] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:16:40 (1786756600) [ 742.883246] LNet: 44031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.891415] LNetError: 44031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.924799] LNet: Removed LNI 192.168.206.7@tcp [ 743.691742] Key type .llcrypt unregistered [ 743.697676] Key type ._llcrypt unregistered [ 749.058070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 749.227222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 749.235762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 749.244167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 751.913856] Key type ._llcrypt registered [ 751.922247] Key type .llcrypt registered [ 752.128205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.141959] alg: No test for adler32 (adler32-zlib) [ 753.094154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.150285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 753.182419] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 753.861519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 753.892279] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 756.067449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.100414] LNet: 44787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.108070] LNetError: 44787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 756.115658] LNet: Removed LNI 10.1.2.3@tcp [ 756.127580] LNet: Removed LNI 192.168.206.7@tcp [ 756.146242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.218333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 756.246638] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 756.253402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 756.713598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.758940] LNet: 44814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.771531] LNetError: 44814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 756.778887] LNetError: 44814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 756.784219] LNet: Removed LNI 10.1.2.3@tcp [ 756.793200] LNet: Removed LNI 192.168.206.7@tcp [ 756.836629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.918949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 756.961573] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 756.972555] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.601439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 757.632032] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 758.888122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.905605] LNet: 44977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.911886] LNetError: 44977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 758.918077] LNetError: 44977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 758.923838] LNet: Removed LNI 10.1.2.3@tcp [ 758.926576] LNet: Removed LNI 192.168.206.7@tcp [ 758.936552] LNet: Removed LNI 192.168.206.7@tcp2 [ 758.977912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.067463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 759.098139] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 759.113852] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.121548] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 767.666852] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:17:05 (1786756625) [ 767.807058] LNet: 45568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.818029] LNetError: 45568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 767.825480] LNetError: 45568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 767.830551] LNet: Removed LNI 10.1.2.3@tcp [ 767.833459] LNet: Removed LNI 192.168.206.7@tcp [ 767.855348] LNet: Removed LNI 192.168.206.7@tcp2 [ 768.447474] Key type .llcrypt unregistered [ 768.451803] Key type ._llcrypt unregistered [ 775.648445] Key type ._llcrypt registered [ 775.652142] Key type .llcrypt registered [ 775.905193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.940307] alg: No test for adler32 (adler32-zlib) [ 776.955556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.999052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 777.047548] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 778.259142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.301253] LNet: 45973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.312216] LNetError: 45973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.335346] LNet: Removed LNI 192.168.206.7@tcp [ 778.357988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.421789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.448092] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 778.765127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.804831] LNet: Removed LNI 192.168.206.7@tcp [ 778.859512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.925194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.959526] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 779.519417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.542797] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.548507] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 779.554909] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.559977] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 779.576463] LNet: Removed LNI 192.168.206.7@tcp [ 779.598406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.694663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 779.721823] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 780.335728] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.469973] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.485441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.494343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.104940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.154499] LNet: 46112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.170829] LNetError: 46112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.207047] LNet: Removed LNI 192.168.206.7@tcp [ 783.227704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.324394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 783.349620] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 790.891904] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:17:28 (1786756648) [ 791.267885] LNet: 46710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.283163] LNetError: 46710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.306281] LNet: Removed LNI 192.168.206.7@tcp [ 792.167513] Key type .llcrypt unregistered [ 792.173369] Key type ._llcrypt unregistered [ 798.222410] Key type ._llcrypt registered [ 798.224980] Key type .llcrypt registered [ 798.432087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.457768] alg: No test for adler32 (adler32-zlib) [ 799.427251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.526459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.577713] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 800.509914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.531768] LNet: 47112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.537229] LNetError: 47112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.554896] LNet: Removed LNI 192.168.206.7@tcp [ 800.569801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.632912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 800.666955] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 807.571730] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:17:45 (1786756665) [ 807.685325] LNet: 47697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.699864] LNetError: 47697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.746345] LNet: Removed LNI 192.168.206.7@tcp [ 808.409536] Key type .llcrypt unregistered [ 808.415830] Key type ._llcrypt unregistered [ 814.842728] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 815.130615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.140619] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.148204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.666307] Key type ._llcrypt registered [ 817.669212] Key type .llcrypt registered [ 817.810259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.821806] alg: No test for adler32 (adler32-zlib) [ 818.733114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.797254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.817518] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 819.535458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.550254] LNet: 48156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.558150] LNetError: 48156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.578375] LNet: Removed LNI 192.168.206.7@tcp [ 819.596381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.656603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 819.681219] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 826.412351] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:18:03 (1786756683) [ 826.551577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.587962] LNet: 48749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.601861] LNetError: 48749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.618606] LNet: Removed LNI 192.168.206.7@tcp [ 826.646710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.755623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 832.529894] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:18:10 (1786756690) [ 832.730722] LNet: 49335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.372229] Key type .llcrypt unregistered [ 833.376938] Key type ._llcrypt unregistered [ 840.318510] Key type ._llcrypt registered [ 840.325849] Key type .llcrypt registered [ 840.660548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.700635] alg: No test for adler32 (adler32-zlib) [ 841.620151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.678427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.705852] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 842.785640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.802826] LNet: 49738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.809421] LNetError: 49738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.823067] LNet: Removed LNI 192.168.206.7@tcp [ 842.855159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.926118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 842.951307] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 850.024632] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:18:27 (1786756707) [ 850.140092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.186380] LNet: 50326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.196515] LNetError: 50326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.219036] LNet: Removed LNI 192.168.206.7@tcp [ 850.240589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.301572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 855.826474] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:18:33 (1786756713) [ 857.480211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 859.305973] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:18:36 (1786756716) [ 860.802041] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 862.375153] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:18:40 (1786756720) [ 863.743706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 865.130430] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:18:43 (1786756723) [ 866.408178] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 868.192346] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:18:45 (1786756725) [ 869.452222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 871.018388] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:18:48 (1786756728) [ 871.093399] LNet: 52376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.757409] Key type .llcrypt unregistered [ 871.762992] Key type ._llcrypt unregistered [ 877.489441] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.599845] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.604716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.609330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.065557] Key type ._llcrypt registered [ 880.067893] Key type .llcrypt registered [ 880.196808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.235234] alg: No test for adler32 (adler32-zlib) [ 881.111552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.176343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.194475] LNet: Added LNI 192.168.206.7@tcp [8/512/0/180] [ 881.198537] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 881.201024] LNet: 52707:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 881.277989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.305542] LNet: 52726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.313829] LNetError: 52726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.318323] LNet: Removed LNI 10.1.2.3@tcp [ 881.320735] LNetError: 52726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.333526] LNet: Removed LNI 192.168.206.7@tcp [ 881.362382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.429810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.453314] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 881.506595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.526985] LNet: 52750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.531959] LNetError: 52750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.542671] LNet: Removed LNI 192.168.206.7@tcp [ 881.567102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.623946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.655382] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 881.662271] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.664617] LNet: 52758:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 881.744543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.765199] LNet: 52776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.769539] LNetError: 52776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.775409] LNet: Removed LNI 10.1.2.3@tcp [ 881.777479] LNetError: 52776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.787284] LNet: Removed LNI 192.168.206.7@tcp [ 881.816552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.885729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.907655] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 881.911029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.913031] LNet: 52784:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 882.060731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.076839] LNet: 52810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.082521] LNetError: 52810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.091750] LNet: Removed LNI 10.1.2.3@tcp [ 882.095094] LNetError: 52810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.106767] LNet: Removed LNI 192.168.206.7@tcp [ 882.129196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.173330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.191860] LNet: Added LNI 192.168.206.7@tcp [8/256/0/182] [ 882.196032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 882.296542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.315534] LNet: 52838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.320933] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.325736] LNet: Removed LNI 10.1.2.3@tcp [ 882.329373] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.348854] LNet: Removed LNI 192.168.206.7@tcp [ 882.376499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.451108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.483500] LNet: Added LNI 192.168.206.7@tcp [8/256/0/182] [ 882.486431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 882.491895] LNet: 52846:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 882.571501] LNet: Removed LNI 10.1.2.3@tcp [ 882.582313] LNet: Removed LNI 192.168.206.7@tcp [ 883.124245] Key type .llcrypt unregistered [ 883.127586] Key type ._llcrypt unregistered [ 891.574547] Key type ._llcrypt registered [ 891.577741] Key type .llcrypt registered [ 892.604525] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:19:10 (1786756750) [ 893.869781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 895.409470] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:19:13 (1786756753) [ 895.718438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 895.735765] alg: No test for adler32 (adler32-zlib) [ 896.615421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.660428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.683438] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 897.358081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.7 [ 897.373600] LNet: 54113:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 897.382685] LNet: Removed LNI 192.168.206.7@tcp [ 902.600886] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:19:20 (1786756760) [ 902.704724] LNet: 54676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.712476] LNetError: 54676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.751568] LNet: Removed LNI 192.168.206.7@tcp [ 903.424199] Key type .llcrypt unregistered [ 903.429138] Key type ._llcrypt unregistered [ 908.607638] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 908.777779] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 911.412207] Key type ._llcrypt registered [ 911.413869] Key type .llcrypt registered [ 911.587568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.614566] alg: No test for adler32 (adler32-zlib) [ 912.963721] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.344061] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:19:36 (1786756776) [ 918.483258] LNet: 55595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.493389] LNetError: 55595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.516595] LNet: Removed LNI 10.1.2.3@tcp [ 919.207411] Key type .llcrypt unregistered [ 919.213099] Key type ._llcrypt unregistered [ 924.882747] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.099732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.104305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 925.108848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 927.924216] Key type ._llcrypt registered [ 927.926178] Key type .llcrypt registered [ 928.164911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.197909] alg: No test for adler32 (adler32-zlib) [ 929.316046] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.324620] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:19:53 (1786756793) [ 937.237987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 939.128605] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:19:56 (1786756796) [ 939.529169] LNet: 56789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.545117] LNetError: 56789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 939.560969] LNet: Removed LNI 10.1.2.3@tcp [ 940.533517] Key type .llcrypt unregistered [ 940.535163] Key type ._llcrypt unregistered [ 947.144191] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 947.444440] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 947.456308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 947.471185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.260725] Key type ._llcrypt registered [ 949.263389] Key type .llcrypt registered [ 949.442563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.497491] alg: No test for adler32 (adler32-zlib) [ 950.439241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.523272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 959.910621] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:20:17 (1786756817) [ 960.509447] LNet: 58014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.516216] LNetError: 58014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 960.531168] LNet: Removed LNI 10.1.2.3@tcp [ 961.016532] Key type .llcrypt unregistered [ 961.023659] Key type ._llcrypt unregistered [ 965.981481] Key type ._llcrypt registered [ 965.983560] Key type .llcrypt registered [ 966.144409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.156635] alg: No test for adler32 (adler32-zlib) [ 967.108335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 967.132109] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 975.031654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 975.661216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_interrupt -r 1 [ 975.703928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_interrupt -r 1 [ 975.756836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 975.787153] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 975.982210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 976.011734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 976.054994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 976.113178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 976.179107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 976.219734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 976.263551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 976.316444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.338699] LNet: 58454:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 976.354599] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 976.359903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.372803] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 976.405958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.435651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.470985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.503365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.542756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.583529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 977.050410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_dropped -r 1 [ 977.081651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_dropped -r 1 [ 977.124034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 977.335851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.365444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 977.385045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.413366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.437075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 977.457996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 977.480766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 977.499919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 977.520450] LNet: 58563:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 977.541205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 977.569632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 977.600677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 977.623664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 977.679672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.717967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 977.754501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 978.230554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_aborted -r 1 [ 978.272583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_aborted -r 1 [ 978.312224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 978.340339] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 978.351700] LNet: Skipped 1 previous similar message [ 978.621425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 978.657860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 978.710391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 978.748393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 978.797120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 978.841777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 978.883670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 978.931637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 978.956437] LNet: 58672:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 978.977146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 979.000514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 979.027973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 979.050849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.079880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.105116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 979.134798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 979.619419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_no_route -r 1 [ 979.655371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_no_route -r 1 [ 979.703469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 979.932656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 979.965376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 980.027140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.082937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.130374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.170165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.213534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.249577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.279781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.311665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.349219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 980.405322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 980.469806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 980.497262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.526738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 981.036450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_timeout -r 1 [ 981.105409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_timeout -r 1 [ 981.159221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 981.194300] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 981.201261] LNet: Skipped 2 previous similar messages [ 981.545155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.585399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 981.640879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.680866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.725757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.782184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 981.831306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.857985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.872565] LNet: 58891:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.878269] LNet: 58891:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 981.898846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.934586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.961620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.996140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 982.049401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.116337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 982.177907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 982.898806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_error -r 1 [ 982.945918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_error -r 1 [ 982.997644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 983.324983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.359777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 983.395497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.430696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.489891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.530534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 983.571837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 983.623492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 983.672681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 983.723992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 983.804211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 983.852869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 983.910602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.978780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 984.033154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 984.314678] LNet: 59045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.320717] LNetError: 59045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.383887] LNet: Removed LNI 192.168.206.7@tcp [ 985.842539] Key type .llcrypt unregistered [ 985.844469] Key type ._llcrypt unregistered [ 1003.163722] Key type ._llcrypt registered [ 1003.169794] Key type .llcrypt registered [ 1004.506345] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:21:02 (1786756862) [ 1005.231320] Key type .llcrypt unregistered [ 1005.233586] Key type ._llcrypt unregistered [ 1011.876392] Key type ._llcrypt registered [ 1011.879382] Key type .llcrypt registered [ 1012.160815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.191903] alg: No test for adler32 (adler32-zlib) [ 1013.156374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1013.190933] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1021.172709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1030.132940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.173302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1030.196364] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1031.391962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_interrupt -r 1 [ 1031.429217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_interrupt -r 1 [ 1031.495597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_interrupt -r 1 [ 1031.536620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_interrupt -r 1 [ 1031.565727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_interrupt -r 1 [ 1031.590532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_interrupt -r 1 [ 1031.627879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_interrupt -r 1 [ 1031.671196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_interrupt -r 1 [ 1031.710494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1031.728765] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1031.733625] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1031.962247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.996852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.029825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.065513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.104434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.144118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.181818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.225579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.246666] LNet: 60517:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.272850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.312661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.347581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.388177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.416832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.445887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.468909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1032.914777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_dropped -r 1 [ 1032.947828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_dropped -r 1 [ 1032.985850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_dropped -r 1 [ 1033.020699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_dropped -r 1 [ 1033.065393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_dropped -r 1 [ 1033.112340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_dropped -r 1 [ 1033.166387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_dropped -r 1 [ 1033.207505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_dropped -r 1 [ 1033.260315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1033.282955] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1033.290578] LNet: Skipped 2 previous similar messages [ 1033.476900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.503156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1033.529378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.568195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.597521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.625757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.646989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.676888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.700782] LNet: 60639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1033.723034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.759408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.790357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.820479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.850574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.883686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.916450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.434344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_aborted -r 1 [ 1034.469024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_aborted -r 1 [ 1034.514935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_aborted -r 1 [ 1034.556445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_aborted -r 1 [ 1034.583472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_aborted -r 1 [ 1034.611741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_aborted -r 1 [ 1034.641810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_aborted -r 1 [ 1034.683630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_aborted -r 1 [ 1034.725198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1034.756164] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1034.763651] LNet: Skipped 2 previous similar messages [ 1034.984877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.015889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1035.049779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.080407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.106532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.139033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.169717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.211694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.254567] LNet: 60760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.278665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.309579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.346928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.381913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.423612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.453958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.496808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.032549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_no_route -r 1 [ 1036.070355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_no_route -r 1 [ 1036.130899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_no_route -r 1 [ 1036.166972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_no_route -r 1 [ 1036.219493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_no_route -r 1 [ 1036.251456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_no_route -r 1 [ 1036.290490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_no_route -r 1 [ 1036.324591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_no_route -r 1 [ 1036.363511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1036.591247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.623309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1036.682991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.736650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.784915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.835471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.870941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.903956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.952205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.984323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.007906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.039073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.088069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.123233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.150376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1037.682755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_timeout -r 1 [ 1037.718305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_timeout -r 1 [ 1037.751518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_timeout -r 1 [ 1037.792388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_timeout -r 1 [ 1037.838404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_timeout -r 1 [ 1037.865680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_timeout -r 1 [ 1037.900559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_timeout -r 1 [ 1037.927433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_timeout -r 1 [ 1037.953538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1037.973645] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1037.978457] LNet: Skipped 5 previous similar messages [ 1038.171580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1038.204871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1038.251993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1038.308346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1038.348614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1038.399687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1038.438723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1038.462959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1038.476587] LNet: 61003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1038.481324] LNet: 61003:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1038.493764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1038.528882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1038.586424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1038.619419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1038.646257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1038.675137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1038.718917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1039.216525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_error -r 1 [ 1039.239578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_error -r 1 [ 1039.265563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_error -r 1 [ 1039.296092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_error -r 1 [ 1039.342714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_error -r 1 [ 1039.388788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_error -r 1 [ 1039.444305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_error -r 1 [ 1039.491218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_error -r 1 [ 1039.525926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1039.794136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.852883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1039.909711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.961229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.014445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.051676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.097913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.153367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.197413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1040.232663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1040.260439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1040.301450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1040.325733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.361345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.394408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1046.533656] LNet: 61204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.539392] LNetError: 61204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.544678] LNet: Removed LNI 192.168.206.7@tcp [ 1047.599703] LNet: Removed LNI 192.168.206.7@tcp1 [ 1048.112721] Key type .llcrypt unregistered [ 1048.117114] Key type ._llcrypt unregistered [ 1063.420425] Key type ._llcrypt registered [ 1063.423683] Key type .llcrypt registered [ 1064.542510] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:22:02 (1786756922) [ 1065.340396] Key type .llcrypt unregistered [ 1065.342708] Key type ._llcrypt unregistered [ 1071.484385] Key type ._llcrypt registered [ 1071.487827] Key type .llcrypt registered [ 1071.734093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1071.749718] alg: No test for adler32 (adler32-zlib) [ 1072.784654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1072.837225] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1080.088265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1080.742897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_dropped -r 1 [ 1080.789480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_dropped -r 1 [ 1080.818608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1080.841235] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1081.052849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.081820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1081.107968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.138751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.199980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.263825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.325846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.398651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.452992] LNet: 62457:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1081.499901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.546957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.602920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.686395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.737979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.794802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.852556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1082.469969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_error -r 1 [ 1082.504264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_error -r 1 [ 1082.533346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1082.551903] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1082.820593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.847498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.868990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.898531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.935506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.969386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.001880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.029593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.051719] LNet: 62566:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.067348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.093376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.122935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.155259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.193177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.234834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.297960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1083.959303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_timeout -r 1 [ 1084.079715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_timeout -r 1 [ 1084.147416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1084.183252] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1084.438469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.475704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.518213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.552378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.581948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.608975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.642954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.667901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.682357] LNet: 62676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.694652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.719816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.754441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.789857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.819284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.854742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.882795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1085.126572] LNet: 62721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.131496] LNetError: 62721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.152789] LNet: Removed LNI 192.168.206.7@tcp [ 1085.611733] Key type .llcrypt unregistered [ 1085.617523] Key type ._llcrypt unregistered [ 1101.467723] Key type ._llcrypt registered [ 1101.470758] Key type .llcrypt registered [ 1102.855217] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:22:40 (1786756960) [ 1103.722335] Key type .llcrypt unregistered [ 1103.724034] Key type ._llcrypt unregistered [ 1110.317393] Key type ._llcrypt registered [ 1110.321486] Key type .llcrypt registered [ 1110.625734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.655911] alg: No test for adler32 (adler32-zlib) [ 1111.604700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1111.626126] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1119.124426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1127.617835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.669270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.695377] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1129.029871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_dropped -r 1 [ 1129.076989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e remote_dropped -r 1 [ 1129.131475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_dropped -r 1 [ 1129.183749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e remote_dropped -r 1 [ 1129.233929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e remote_dropped -r 1 [ 1129.294122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e remote_dropped -r 1 [ 1129.354451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e remote_dropped -r 1 [ 1129.406284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e remote_dropped -r 1 [ 1129.444885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1129.461962] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1129.472382] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.107@tcp [ 1129.723251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.768410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.794195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.848362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.888453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.926465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.964920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.004569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.024308] LNet: 64192:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.040769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.070573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.096549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.155992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.195361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.229507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.255227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.286311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.901403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_error -r 1 [ 1130.956548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e remote_error -r 1 [ 1131.009378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_error -r 1 [ 1131.092684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e remote_error -r 1 [ 1131.148877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e remote_error -r 1 [ 1131.206502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e remote_error -r 1 [ 1131.252965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e remote_error -r 1 [ 1131.310701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e remote_error -r 1 [ 1131.388828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1131.414602] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1131.421681] LNet: Skipped 2 previous similar messages [ 1131.777894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.834691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.876654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.915897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.945201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.979547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.033546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.068525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.090356] LNet: 64315:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.108246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.145214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.170888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.209952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.239177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.312565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.347458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1132.427607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.933460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_timeout -r 1 [ 1132.966966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e remote_timeout -r 1 [ 1133.000914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_timeout -r 1 [ 1133.036259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e remote_timeout -r 1 [ 1133.070905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e remote_timeout -r 1 [ 1133.110448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e remote_timeout -r 1 [ 1133.141882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e remote_timeout -r 1 [ 1133.169443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e remote_timeout -r 1 [ 1133.200375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.107@tcp [ 1133.220829] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1133.475754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.515984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.550121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.589529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.673866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.753280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.794538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.817376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.831411] LNet: 64438:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.859430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.895484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.941958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.998537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.029313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.076790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.136968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.210169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.964907] LNet: 64520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.973677] LNetError: 64520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.980909] LNet: Removed LNI 192.168.206.7@tcp [ 1138.988659] LNet: Removed LNI 192.168.206.7@tcp1 [ 1139.293090] Key type .llcrypt unregistered [ 1139.297920] Key type ._llcrypt unregistered [ 1155.260540] Key type ._llcrypt registered [ 1155.262312] Key type .llcrypt registered [ 1156.385866] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:23:34 (1786757014) [ 1157.764147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1159.158259] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:23:37 (1786757017) [ 1159.952292] Key type .llcrypt unregistered [ 1159.956336] Key type ._llcrypt unregistered [ 1165.573742] Key type ._llcrypt registered [ 1165.578948] Key type .llcrypt registered [ 1165.847542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1165.880626] alg: No test for adler32 (adler32-zlib) [ 1166.812423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1166.846833] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1173.976855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1174.528618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e network_timeout -r 1 [ 1174.567046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e network_timeout -r 1 [ 1174.602971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1174.624461] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1174.817912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.843209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.871785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.893787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.916559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.940250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.963663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.994301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.024694] LNet: 66066:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.041690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.067714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.072508] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1175.094868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.130422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.165481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.199693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.442683] LNet: 66109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.449071] LNetError: 66109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.489229] LNet: Removed LNI 192.168.206.7@tcp [ 1176.825464] Key type .llcrypt unregistered [ 1176.831140] Key type ._llcrypt unregistered [ 1195.788423] Key type ._llcrypt registered [ 1195.791802] Key type .llcrypt registered [ 1196.013307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1196.024890] alg: No test for adler32 (adler32-zlib) [ 1197.007928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1197.029036] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1204.282195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1212.849572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.928829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.971079] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1214.355923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e network_timeout -r 1 [ 1214.383660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e network_timeout -r 1 [ 1214.412468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e network_timeout -r 1 [ 1214.446438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e network_timeout -r 1 [ 1214.485510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e network_timeout -r 1 [ 1214.535433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e network_timeout -r 1 [ 1214.572636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e network_timeout -r 1 [ 1214.608825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e network_timeout -r 1 [ 1214.650691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1214.670953] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1214.895909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.947567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.987762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.019890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.053776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.087750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.121923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.163593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.201849] LNet: 66993:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.224958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.273332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.327498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.358675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.420772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.461248] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1215.464234] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1215.469902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.504386] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1220.186110] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1220.569621] LNet: 67071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.575581] LNetError: 67071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.582868] LNet: Removed LNI 192.168.206.7@tcp [ 1220.600222] LNet: Removed LNI 192.168.206.7@tcp1 [ 1221.022495] Key type .llcrypt unregistered [ 1221.028219] Key type ._llcrypt unregistered [ 1235.739303] Key type ._llcrypt registered [ 1235.747927] Key type .llcrypt registered [ 1236.895745] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:24:54 (1786757094) [ 1237.269337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1237.293420] alg: No test for adler32 (adler32-zlib) [ 1238.279441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.351817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.396395] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1239.121968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1239.135066] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1239.817797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1239.898619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1239.963537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1240.022059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1240.086449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1240.150663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1240.204779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.7@tcp [ 1240.225242] LNetError: 68170:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.7@tcp to recovery [ 1240.353161] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1240.353271] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1242.402956] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1242.673668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1246.496447] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1247.346486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1247.388711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.488527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1247.516862] LNet: 68277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.532582] LNetError: 68277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.541762] LNet: Removed LNI 192.168.206.7@tcp [ 1247.556171] LNet: Removed LNI 192.168.206.7@tcp1 [ 1247.580852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.634586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1247.659711] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1248.186480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1248.202516] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1248.910933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1248.949697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.005351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1249.051913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1249.090953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1249.127874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1249.185334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1249.222771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.7@tcp [ 1249.242776] LNetError: 68549:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.7@tcp to recovery [ 1249.696385] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1252.773777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1255.842837] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1255.850319] LNet: Skipped 1 previous similar message [ 1260.699361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1260.740960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.768652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.621402] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:25:24 (1786757124) [ 1267.092425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1267.125478] LNet: 69266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.135694] LNetError: 69266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.141574] LNetError: 69266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1267.150774] LNet: Removed LNI 192.168.206.7@tcp [ 1267.170026] LNet: Removed LNI 192.168.206.7@tcp1 [ 1267.217469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.307303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1267.343162] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1268.003746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1268.024755] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1269.002666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1269.077681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1269.113725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1269.161710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1269.191467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1269.218931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1269.246233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.7@tcp --health 0 [ 1269.348994] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.7@tcp [ 1269.392734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1270.369480] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1270.380054] LNet: Skipped 1 previous similar message [ 1284.797229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1284.924621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1284.946649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.976750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.7@tcp --health 500 [ 1285.061389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1291.573209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1291.636589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.674802] LNet: 69757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.689674] LNetError: 69757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.698767] LNetError: 69757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1291.705371] LNet: Removed LNI 192.168.206.7@tcp [ 1291.717814] LNet: Removed LNI 192.168.206.7@tcp1 [ 1291.735690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.806830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.835276] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1292.471584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1292.497923] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1293.225747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1293.256353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.311510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1293.330526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1293.354415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1293.391282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1293.418113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1293.464832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.7@tcp --health 0 [ 1294.819331] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1294.828254] LNet: Skipped 3 previous similar messages [ 1296.030906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1305.214632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1305.266086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.320586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.468218] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:26:10 (1786757170) [ 1313.160183] LNet: 70803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.164386] LNetError: 70803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.168224] LNetError: 70803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1313.172322] LNet: Removed LNI 192.168.206.7@tcp [ 1313.183693] LNet: Removed LNI 192.168.206.7@tcp1 [ 1313.817399] Key type .llcrypt unregistered [ 1313.821488] Key type ._llcrypt unregistered [ 1321.823396] Key type ._llcrypt registered [ 1321.828225] Key type .llcrypt registered [ 1322.081249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.094212] alg: No test for adler32 (adler32-zlib) [ 1323.196966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1323.279780] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1334.109571] Lustre: DEBUG MARKER: Initial discovery [ 1334.202858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.107@tcp [ 1336.168355] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1336.288442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1336.315947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.107@tcp [ 1336.335200] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1336.347069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1336.376658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.107@tcp -r 1 -m GET -l 3 [ 1336.405043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.107@tcp -r 1 -m GET -e local_error [ 1336.445963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.107@tcp -r 1 -m PUT -l 6 [ 1336.611588] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1337.911332] Lustre: DEBUG MARKER: Force oleg607-client.virtnet to discover 192.168.206.107@tcp (in background) [ 1338.163951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.107@tcp [ 1339.517192] Lustre: DEBUG MARKER: Force oleg607-server to discover 192.168.206.7@tcp [ 1341.216475] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1342.153867] Lustre: DEBUG MARKER: Wait for 71584 [ 1343.615674] Lustre: DEBUG MARKER: Finished wait on 71584 [ 1343.671461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1343.727043] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1343.739276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.116159] LNet: 72037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.131820] LNetError: 72037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.149131] LNet: Removed LNI 192.168.206.7@tcp [ 1344.820519] Key type .llcrypt unregistered [ 1344.823301] Key type ._llcrypt unregistered [ 1364.820981] Key type ._llcrypt registered [ 1364.823246] Key type .llcrypt registered [ 1366.084547] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:27:03 (1786757223) [ 1366.607423] Key type .llcrypt unregistered [ 1366.609366] Key type ._llcrypt unregistered [ 1372.091514] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1372.243907] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1372.247017] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1372.252018] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1374.919455] Key type ._llcrypt registered [ 1374.923614] Key type .llcrypt registered [ 1375.067925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1375.077358] alg: No test for adler32 (adler32-zlib) [ 1375.961326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.023711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1376.053487] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1376.882593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1376.906772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1385.664470] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:27:23 (1786757243) [ 1385.826814] LNet: 74114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.832575] LNetError: 74114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1385.851144] LNet: Removed LNI 10.1.2.3@tcp [ 1385.875333] LNet: Removed LNI 192.168.206.7@tcp [ 1386.861465] Key type .llcrypt unregistered [ 1386.863090] Key type ._llcrypt unregistered [ 1394.605909] Key type ._llcrypt registered [ 1394.608720] Key type .llcrypt registered [ 1394.785561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1394.803296] alg: No test for adler32 (adler32-zlib) [ 1395.659916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.702348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1395.720589] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1396.582560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1396.609914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1397.368845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.7@tcp [ 1398.432157] LNetError: Unexpected error -22 connecting to 192.168.206.7@tcp at host 192.168.206.7:988 [ 1403.409357] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:27:41 (1786757261) [ 1403.638704] LNet: 75255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.650320] LNetError: 75255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1403.659273] LNet: Removed LNI 10.1.2.3@tcp [ 1403.675850] LNet: Removed LNI 192.168.206.7@tcp [ 1404.514560] Key type .llcrypt unregistered [ 1404.520312] Key type ._llcrypt unregistered [ 1410.719570] Key type ._llcrypt registered [ 1410.723593] Key type .llcrypt registered [ 1411.011223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1411.050952] alg: No test for adler32 (adler32-zlib) [ 1411.996962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.067759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1412.115534] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1413.063862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1413.110605] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 1414.140745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.7@tcp1 --nid 192.168.206.7@tcp2 [ 1418.131667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp1 192.168.206.7@tcp2 [ 1418.171472] LNetError: 76095:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.7@tcp1 to 192.168.206.7@tcp2 - Route table may be misconfigured [ 1418.209634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp2 192.168.206.7@tcp1 [ 1425.182681] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:28:03 (1786757283) [ 1425.483897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.514859] LNet: 76668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.537979] LNetError: 76668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.552589] LNet: Removed LNI 192.168.206.7@tcp1 [ 1426.604486] LNet: Removed LNI 192.168.206.7@tcp2 [ 1426.644285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1426.723500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1426.749221] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1427.582967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1427.614727] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1428.317568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1428.364389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.7@tcp [ 1428.400575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp1 192.168.206.7@tcp1 [ 1428.446773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.7@tcp -d 192.168.206.7@tcp -e network_timeout [ 1428.480874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -e network_timeout [ 1428.509974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -e network_timeout [ 1428.538895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -e network_timeout [ 1428.565271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1428.580422] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1428.596248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.677624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.763162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1428.769130] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1434.159307] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:28:12 (1786757292) [ 1434.237813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.296392] LNet: 77522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.313449] LNetError: 77522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.319658] LNetError: 77522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1434.333254] LNet: Removed LNI 192.168.206.7@tcp [ 1435.370396] LNet: Removed LNI 192.168.206.7@tcp1 [ 1435.385882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.434396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1435.721110] LNet: 77572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.088121] Key type .llcrypt unregistered [ 1436.091103] Key type ._llcrypt unregistered [ 1446.683920] Key type ._llcrypt registered [ 1446.686512] Key type .llcrypt registered [ 1447.626326] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:28:25 (1786757305) [ 1448.028982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.045019] alg: No test for adler32 (adler32-zlib) [ 1448.923694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.065150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1451.604348] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:28:29 (1786757309) [ 1451.698191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.712159] LNet: 78662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.732618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.808183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.837607] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1452.501543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.534959] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1453.368635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1453.396178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp1 [ 1453.420975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp1 [ 1458.721269] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:28:36 (1786757316) [ 1460.304413] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg607-server [ 1461.883310] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:28:39 (1786757319) [ 1463.943808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg607-server [ 1465.627649] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:28:43 (1786757323) [ 1467.625117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg607-server [ 1469.124597] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:28:47 (1786757327) [ 1470.697603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg607-server [ 1472.460964] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:28:50 (1786757330) [ 1474.140507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg607-server [ 1475.700478] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:28:53 (1786757333) [ 1477.325186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg607-server [ 1478.926744] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:28:56 (1786757336) [ 1480.494987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg607-server [ 1481.942173] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:29:00 (1786757340) [ 1483.610599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg607-server [ 1485.224129] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:29:03 (1786757343) [ 1487.051941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg607-server [ 1488.546710] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:29:06 (1786757346) [ 1488.678761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.739372] LNet: 82540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.758845] LNetError: 82540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.765982] LNet: Removed LNI 192.168.206.7@tcp [ 1489.831542] LNet: Removed LNI 192.168.206.7@tcp1 [ 1489.857448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.912959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1489.930094] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1490.677418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1490.766960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1490.955882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.978893] LNet: 82693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.984605] LNetError: 82693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.987184] LNetError: 82693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.001676] LNet: Removed LNI 192.168.206.7@tcp [ 1491.015752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.054862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1491.069119] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1491.706498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1491.870421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.975549] LNet: Removed LNI 192.168.206.7@tcp [ 1493.002186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.052701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.070104] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1493.634625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1493.699041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1493.851361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.886967] LNet: 82985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.891427] LNet: 82985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1493.896365] LNetError: 82985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.902052] LNetError: 82985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1493.923081] LNet: Removed LNI 192.168.206.7@tcp [ 1493.954410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.018530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1494.043738] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1494.878356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1496.237419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1496.263242] LNetError: 83140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.320160] LNet: Removed LNI 192.168.206.7@tcp [ 1497.370898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.430682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.448394] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1498.042830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1498.117383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1499.542205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.555164] LNet: 83296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.562188] LNet: 83296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1500.582448] LNet: Removed LNI 192.168.206.7@tcp [ 1500.621197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.676574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1500.695065] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1501.419604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1501.658318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.684954] LNetError: 83441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.698949] LNetError: 83441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1501.722418] LNet: Removed LNI 192.168.206.7@tcp [ 1501.765565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.844691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.886885] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1502.797221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1502.880149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1503.172257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.234438] LNet: Removed LNI 192.168.206.7@tcp [ 1504.264348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.349990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1504.377292] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1505.149736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1505.371283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.475534] LNet: Removed LNI 192.168.206.7@tcp [ 1506.532403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.612339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1506.635798] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1507.390674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1507.458317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1507.726735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.755170] LNet: 83881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.760847] LNet: 83881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1508.849567] LNet: Removed LNI 192.168.206.7@tcp [ 1508.879780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.949451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1508.984775] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1509.845226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1511.402953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.438880] LNetError: 84036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.455250] LNetError: 84036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1511.468760] LNet: Removed LNI 192.168.206.7@tcp [ 1511.488509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.565704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.602594] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1512.445512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1512.525758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1512.819702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.894427] LNet: Removed LNI 192.168.206.7@tcp [ 1513.952688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.052340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1514.109424] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1514.820367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1515.242189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.332912] LNet: Removed LNI 192.168.206.7@tcp [ 1516.368218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.450327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.477922] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1517.107648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1517.143502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.178503] LNet: Removed LNI 192.168.206.7@tcp [ 1517.201689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.249496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.268050] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1517.864712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1517.923837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.944890] LNet: Removed LNI 192.168.206.7@tcp [ 1517.963959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.015405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1518.092565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.123146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.186872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1518.275549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.315573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.380826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1518.484641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.535701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.598778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1518.628291] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1519.344681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.381032] LNet: Removed LNI 192.168.206.7@tcp [ 1519.403388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.486186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1525.755377] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:29:43 (1786757383) [ 1526.007415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.060655] LNet: 85349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.070715] LNet: 85349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1526.149362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.220100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1526.277966] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/180] [ 1527.629981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.659706] LNetError: 85487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.664804] LNetError: 85487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1527.679685] LNet: Removed LNI 192.168.206.7@tcp231 [ 1527.701102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.807306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1527.847610] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/180] [ 1527.855988] LNet: 85495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1528.402787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1528.437735] LNet: Removed LNI 192.168.206.7@tcp231 [ 1528.455789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1528.494469] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/180] [ 1529.590933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.626315] LNet: Removed LNI 192.168.206.7@tcp231 [ 1529.649273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.733780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1529.765864] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/0] [ 1529.773472] LNet: 85647:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1530.143902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1530.199314] LNet: Removed LNI 192.168.206.7@tcp231 [ 1530.223731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1530.269675] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/0] [ 1531.378770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.399854] LNet: Removed LNI 192.168.206.7@tcp231 [ 1531.424129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.501752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1531.540605] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/60] [ 1531.545643] LNet: 85799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1532.040687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1532.096009] LNet: Removed LNI 192.168.206.7@tcp231 [ 1532.119309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1532.165179] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/60] [ 1533.239293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.272551] LNet: Removed LNI 192.168.206.7@tcp231 [ 1533.289309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.373577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1533.397257] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/180] [ 1533.834749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1533.862717] LNet: Removed LNI 192.168.206.7@tcp231 [ 1533.880970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1533.924764] LNet: Added LNI 192.168.206.7@tcp231 [8/256/0/180] [ 1540.623703] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:29:58 (1786757398) [ 1540.708874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.770191] LNet: Removed LNI 192.168.206.7@tcp231 [ 1540.811923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.910535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1540.952222] LNet: Added LNI 192.168.206.7@tcp232 [8/256/0/180] [ 1541.603245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1541.644780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1541.707484] LNet: Removed LNI 192.168.206.7@tcp232 [ 1541.719597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1541.744221] LNet: Added LNI 192.168.206.7@tcp232 [8/256/0/180] [ 1542.383905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1542.432885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1542.468395] LNet: Removed LNI 192.168.206.7@tcp232 [ 1542.486780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1542.508043] LNet: Added LNI 192.168.206.7@tcp232 [8/256/0/180] [ 1543.204181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1543.262061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1543.302593] LNet: Removed LNI 192.168.206.7@tcp232 [ 1543.315177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1543.357546] LNet: Added LNI 192.168.206.7@tcp232 [8/256/0/180] [ 1543.951644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1544.001293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1544.027365] LNet: Removed LNI 192.168.206.7@tcp232 [ 1544.050248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1549.419457] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:30:07 (1786757407) [ 1550.322489] Key type .llcrypt unregistered [ 1550.324942] Key type ._llcrypt unregistered [ 1555.785789] Key type ._llcrypt registered [ 1555.788129] Key type .llcrypt registered [ 1555.961564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1555.980379] alg: No test for adler32 (adler32-zlib) [ 1556.839660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1556.860722] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1563.080910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1573.418217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.482707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1573.526820] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1574.889534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_interrupt -r 2 [ 1574.940803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_interrupt -r 2 [ 1574.990640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_interrupt -r 2 [ 1575.030948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_interrupt -r 2 [ 1575.065811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_interrupt -r 2 [ 1575.094618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_interrupt -r 2 [ 1575.136526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_interrupt -r 2 [ 1575.182631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_interrupt -r 2 [ 1575.210988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.107@tcp [ 1575.235302] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1575.244834] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1575.526846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1575.558416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1575.590909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1575.624939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1575.655829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1575.682213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1575.709568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1575.739747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1575.765507] LNet: 88453:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1575.787479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1575.831580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1575.850612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1575.891890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1575.933220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1575.958194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1576.455573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_dropped -r 2 [ 1576.509904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_dropped -r 2 [ 1576.557436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_dropped -r 2 [ 1576.588650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_dropped -r 2 [ 1576.629700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_dropped -r 2 [ 1576.689755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_dropped -r 2 [ 1576.760735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_dropped -r 2 [ 1576.808482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_dropped -r 2 [ 1576.852866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.107@tcp [ 1576.866684] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1577.120845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1577.151521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1577.183519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1577.211754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1577.247151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1577.306482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1577.360347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1577.404948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1577.421952] LNet: 88573:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.436333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.468340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.515288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1577.567222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1577.611232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1577.664067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1578.164325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_aborted -r 2 [ 1578.197948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_aborted -r 2 [ 1578.224232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_aborted -r 2 [ 1578.251338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_aborted -r 2 [ 1578.285083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_aborted -r 2 [ 1578.312943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_aborted -r 2 [ 1578.340874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_aborted -r 2 [ 1578.375515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_aborted -r 2 [ 1578.415554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.107@tcp [ 1578.436412] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1578.675844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.697743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1578.722446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.742717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.770755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1578.798850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1578.821304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.843330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1578.867890] LNet: 88693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1578.887950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1578.917817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1578.952893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.983711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.031743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.057591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.515423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_no_route -r 2 [ 1579.550991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_no_route -r 2 [ 1579.614646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_no_route -r 2 [ 1579.664498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_no_route -r 2 [ 1579.712272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_no_route -r 2 [ 1579.745593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_no_route -r 2 [ 1579.773782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_no_route -r 2 [ 1579.809866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_no_route -r 2 [ 1579.856619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.107@tcp [ 1580.077356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1580.118724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1580.182142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1580.219526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1580.244994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1580.274889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1580.298689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1580.334525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1580.364593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1580.395632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1580.424918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1580.457549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1580.487917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1580.514476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1580.957936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e local_timeout -r 2 [ 1580.990918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e local_timeout -r 2 [ 1581.021871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e local_timeout -r 2 [ 1581.075588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e local_timeout -r 2 [ 1581.103927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e local_timeout -r 2 [ 1581.130754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e local_timeout -r 2 [ 1581.160795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e local_timeout -r 2 [ 1581.198480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e local_timeout -r 2 [ 1581.239820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.107@tcp [ 1581.267550] LNet: There was an unexpected network error while writing to 192.168.206.107: rc = -22 [ 1581.272426] LNet: Skipped 1 previous similar message [ 1581.469900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1581.495416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1581.518402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1581.540296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1581.567907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1581.592708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1581.618934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1581.644384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1581.658726] LNet: 88934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.663499] LNet: 88934:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1581.677856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1581.702043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1581.731493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1581.795741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.821857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1581.854993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1582.287791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp -m GET -e remote_dropped -r 2 [ 1582.328717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp -d 192.168.206.107@tcp1 -m GET -e remote_dropped -r 2 [ 1582.362880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp -m GET -e remote_dropped -r 2 [ 1582.431622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp -d 192.168.206.7@tcp1 -m GET -e remote_dropped -r 2 [ 1582.480875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp -m GET -e remote_dropped -r 2 [ 1582.522256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.107@tcp1 -m GET -e remote_dropped -r 2 [ 1582.555382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp -m GET -e remote_dropped -r 2 [ 1582.590471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.7@tcp1 -d 192.168.206.7@tcp1 -m GET -e remote_dropped -r 2 [ 1582.635705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.7@tcp 192.168.206.107@tcp [ 1582.660550] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.107@tcp [ 1582.865921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1582.893051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1582.928693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1582.959424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1582.986597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1583.017765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1583.054781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1583.090835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1583.123485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1583.151765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1583.182359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1583.206806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1583.235489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1583.278815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1587.825506] LNet: 89132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.833770] LNetError: 89132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.838640] LNet: Removed LNI 192.168.206.7@tcp [ 1587.852409] LNet: Removed LNI 192.168.206.7@tcp1 [ 1588.268704] Key type .llcrypt unregistered [ 1588.271598] Key type ._llcrypt unregistered [ 1602.638768] Key type ._llcrypt registered [ 1602.642549] Key type .llcrypt registered [ 1603.671969] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:31:01 (1786757461) [ 1604.030535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.048726] alg: No test for adler32 (adler32-zlib) [ 1604.959838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.025467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.065509] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1605.810570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1605.831819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.7@tcp [ 1605.847163] LNetError: 90109:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.7@tcp to recovery [ 1605.915767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.7@tcp [ 1606.014706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1611.406692] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:31:09 (1786757469) [ 1611.504984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.524121] LNet: 90696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.536366] LNetError: 90696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.579140] LNet: Removed LNI 192.168.206.7@tcp [ 1611.606181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.654228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.690685] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1612.496212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1612.529134] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1613.545585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.7@tcp [ 1613.573175] LNetError: 90965:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.7@tcp to recovery [ 1613.673643] LNet: 1 local NIs in recovery (showing 1): 192.168.206.7@tcp [ 1613.674907] LNet: There was an unexpected network error while writing to 192.168.206.7: rc = -22 [ 1614.932641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.7@tcp [ 1622.166316] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:31:20 (1786757480) [ 1624.111081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg607-server [ 1626.189874] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:31:23 (1786757483) [ 1628.330995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg607-server [ 1629.825195] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:31:27 (1786757487) [ 1631.897228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg607-server [ 1634.882291] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:31:31 (1786757491) [ 1635.003139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.030774] LNet: 92579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.044723] LNetError: 92579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.064658] LNet: Removed LNI 192.168.206.7@tcp [ 1635.105985] LNet: Removed LNI 192.168.206.7@tcp1 [ 1635.145656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.235982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.306554] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1636.258617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1636.341605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1636.385446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1636.482671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1636.551194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1636.584445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1642.950414] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:31:40 (1786757500) [ 1643.088542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.126040] LNet: 93310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.139937] LNetError: 93310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.153541] LNetError: 93310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1643.182688] LNet: Removed LNI 192.168.206.7@tcp [ 1643.204334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.263641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1643.306129] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1649.484436] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:31:47 (1786757507) [ 1649.680799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.731946] LNet: 94014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.749637] LNetError: 94014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.777318] LNet: Removed LNI 192.168.206.7@tcp [ 1649.795509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.850937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.876652] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1656.318714] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:31:54 (1786757514) [ 1657.923895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1659.500265] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:31:57 (1786757517) [ 1660.119299] LNet: 95050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.134120] LNetError: 95050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.152243] LNet: Removed LNI 192.168.206.7@tcp [ 1660.814601] Key type .llcrypt unregistered [ 1660.816915] Key type ._llcrypt unregistered [ 1666.616510] Key type ._llcrypt registered [ 1666.620439] Key type .llcrypt registered [ 1666.852278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.880037] alg: No test for adler32 (adler32-zlib) [ 1667.784356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1667.806150] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1674.411605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1678.152310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.107@tcp [ 1678.439943] LNet: 95475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.448844] LNetError: 95475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.460762] LNet: Removed LNI 192.168.206.7@tcp [ 1678.793864] Key type .llcrypt unregistered [ 1678.796977] Key type ._llcrypt unregistered [ 1687.758805] Key type ._llcrypt registered [ 1687.767733] Key type .llcrypt registered [ 1688.984401] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1690.429801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1691.825323] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:32:29 (1786757549) [ 1693.166346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1694.554616] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:32:32 (1786757552) [ 1695.848700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg607-server [ 1696.987877] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:32:35 (1786757555) [ 1698.312841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg607-server [ 1699.860678] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:32:37 (1786757557) [ 1700.900778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.936855] alg: No test for adler32 (adler32-zlib) [ 1701.976489] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1702.190517] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1703.824228] Key type lgssc registered [ 1704.785807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1721.122910] Key type lgssc unregistered [ 1721.334754] LNet: 98651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.339982] LNetError: 98651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.357644] LNet: Removed LNI 192.168.206.7@tcp [ 1721.952813] Key type .llcrypt unregistered [ 1721.959727] Key type ._llcrypt unregistered [ 1734.414471] Key type ._llcrypt registered [ 1734.416474] Key type .llcrypt registered [ 1735.909485] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:33:13 (1786757593) [ 1736.592110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.619689] alg: No test for adler32 (adler32-zlib) [ 1737.651375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.738618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1737.813670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1737.886202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1737.957056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1745.144786] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:33:22 (1786757602) [ 1745.319058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.363058] LNet: 100062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.431477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.565689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.600588] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1746.945715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.972602] LNet: 100214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.983807] LNetError: 100214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.009763] LNet: Removed LNI 192.168.206.7@tcp [ 1747.062984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.167651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1747.226210] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1748.077362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.096269] LNet: 100242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.101721] LNetError: 100242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.115717] LNet: Removed LNI 192.168.206.7@tcp [ 1748.143116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.223432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1748.280499] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1754.297778] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:33:32 (1786757612) [ 1754.422056] LNet: 100823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.436531] LNetError: 100823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.471814] LNet: Removed LNI 192.168.206.7@tcp [ 1755.132458] Key type .llcrypt unregistered [ 1755.134422] Key type ._llcrypt unregistered [ 1761.074993] Key type ._llcrypt registered [ 1761.078887] Key type .llcrypt registered [ 1761.302467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1761.329474] alg: No test for adler32 (adler32-zlib) [ 1762.203241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.274432] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1762.281043] LNet: *** cfs_fail_loc=e003, val=100*** [ 1762.897465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1762.920948] LNet: *** cfs_fail_loc=e003, val=100*** [ 1762.925034] LNet: Skipped 2 previous similar messages [ 1762.929041] Ping reply received with mapping: [ 1762.931060] found metadata reply NI[192.168.206.7@tcp] [ 1762.934691] 07 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1762.939328] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1762.945190] 5a 5a 5a 5a 5a 5a 5a 5a [ 1763.054863] LNet: 101234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.060253] LNetError: 101234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.073028] LNet: Removed LNI 192.168.206.7@tcp [ 1764.827560] Key type .llcrypt unregistered [ 1764.831801] Key type ._llcrypt unregistered [ 1774.965281] Key type ._llcrypt registered [ 1774.967331] Key type .llcrypt registered [ 1775.962757] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:33:54 (1786757634) [ 1776.641260] Key type .llcrypt unregistered [ 1776.646259] Key type ._llcrypt unregistered [ 1784.094210] Key type ._llcrypt registered [ 1784.098593] Key type .llcrypt registered [ 1784.322617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1784.346624] alg: No test for adler32 (adler32-zlib) [ 1785.464608] LNetError: 102292:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1785.825330] Key type .llcrypt unregistered [ 1785.831669] Key type ._llcrypt unregistered [ 1791.667812] Key type ._llcrypt registered [ 1791.669733] Key type .llcrypt registered [ 1793.795157] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:34:10 (1786757650) [ 1794.827970] Key type .llcrypt unregistered [ 1794.834452] Key type ._llcrypt unregistered [ 1801.263196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1801.271744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1803.956837] Key type ._llcrypt registered [ 1803.958793] Key type .llcrypt registered [ 1804.142310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1804.160159] alg: No test for adler32 (adler32-zlib) [ 1805.166451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.243711] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1815.554550] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:34:32 (1786757672) [ 1815.604024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.630533] LNet: 104075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.645829] LNetError: 104075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1815.668902] LNet: Removed LNI 10.1.2.3@tcp [ 1815.683632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.638623] Key type .llcrypt unregistered [ 1816.641707] Key type ._llcrypt unregistered [ 1822.371121] Key type ._llcrypt registered [ 1822.374428] Key type .llcrypt registered [ 1822.677459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1822.703209] alg: No test for adler32 (adler32-zlib) [ 1823.605609] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1823.670131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1823.783636] LNet: 104373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.792989] LNetError: 104373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.808235] LNet: Removed LNI 192.168.206.7@tcp [ 1824.411472] Key type .llcrypt unregistered [ 1824.413139] Key type ._llcrypt unregistered [ 1834.944384] Key type ._llcrypt registered [ 1834.946751] Key type .llcrypt registered [ 1836.277332] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:34:53 (1786757693) [ 1837.369271] Key type .llcrypt unregistered [ 1837.370958] Key type ._llcrypt unregistered [ 1844.317559] Key type ._llcrypt registered [ 1844.321733] Key type .llcrypt registered [ 1844.481510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1844.493214] alg: No test for adler32 (adler32-zlib) [ 1853.488740] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:35:11 (1786757711) [ 1853.712116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.785965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.874836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1853.906442] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1854.413736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1854.440349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1854.474688] LNet: Removed LNI 192.168.206.7@tcp [ 1854.803477] LNet: 106130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.277374] Key type .llcrypt unregistered [ 1855.281493] Key type ._llcrypt unregistered [ 1865.874811] Key type ._llcrypt registered [ 1865.883219] Key type .llcrypt registered [ 1867.019864] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:35:24 (1786757724) [ 1867.344292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1867.360621] alg: No test for adler32 (adler32-zlib) [ 1868.290085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.364754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.403439] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1869.303232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1869.355345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.7@tcp [ 1869.370371] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1869.374692] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1869.376737] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1869, 0, 0 [ 1869.379941] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1869.382246] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1869.384870] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1869.390713] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1869, 0, 0 [ 1869.397348] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1869.399679] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1869.402713] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1869.404740] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1869, 0, 1919 [ 1869.411419] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1869.414267] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1869.417159] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1869.421980] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1869, 0, 1919 [ 1869.426879] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1874.774824] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:35:32 (1786757732) [ 1875.252432] LNet: 107671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.263650] LNetError: 107671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.277905] LNet: Removed LNI 192.168.206.7@tcp [ 1875.850492] Key type .llcrypt unregistered [ 1875.852215] Key type ._llcrypt unregistered [ 1880.742270] Key type ._llcrypt registered [ 1880.743787] Key type .llcrypt registered [ 1880.894613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.906663] alg: No test for adler32 (adler32-zlib) [ 1881.774579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1881.810074] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1887.840838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 1894.950413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.978272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1895.000298] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 1895.758193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1895.905773] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1895.913855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1895.918189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1898.200313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1898.219217] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1907.284180] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:36:04 (1786757764) [ 1907.389991] LNet: 109175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.397788] LNetError: 109175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.409741] LNet: Removed LNI 192.168.206.7@tcp [ 1907.416046] LNet: Removed LNI 192.168.206.7@tcp1 [ 1907.436137] LNet: Removed LNI 10.1.2.3@tcp99 [ 1908.073756] Key type .llcrypt unregistered [ 1908.076862] Key type ._llcrypt unregistered [ 1913.723504] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1913.738180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1916.350449] Key type ._llcrypt registered [ 1916.353182] Key type .llcrypt registered [ 1916.567050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1916.589834] alg: No test for adler32 (adler32-zlib) [ 1917.538722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.589768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.610402] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1918.143587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1918.159307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1919.911686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1919.947514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1920.005930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1920.030493] LNetError: 109931:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1920.073766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1920.110057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1920.469673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1920.530834] LNet: 109951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.535600] LNetError: 109951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1920.541325] LNet: Removed LNI 10.1.2.3@tcp [ 1921.574514] LNet: Removed LNI 192.168.206.7@tcp [ 1922.215520] Key type .llcrypt unregistered [ 1922.218802] Key type ._llcrypt unregistered [ 1932.562670] Key type ._llcrypt registered [ 1932.564249] Key type .llcrypt registered [ 1933.677961] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:36:31 (1786757791) [ 1934.107179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.118886] alg: No test for adler32 (adler32-zlib) [ 1935.096585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.170735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1935.197685] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1935.900370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 1935.983917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg607-client@tcp [ 1941.658927] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:36:39 (1786757799) [ 1941.773222] LNet: 111479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.782662] LNetError: 111479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.811863] LNet: Removed LNI 192.168.206.7@tcp [ 1942.476608] Key type .llcrypt unregistered [ 1942.480387] Key type ._llcrypt unregistered [ 1948.180837] Key type ._llcrypt registered [ 1948.182874] Key type .llcrypt registered [ 1948.291967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.312562] alg: No test for adler32 (adler32-zlib) [ 1949.208379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1949.231271] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1949.257325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1954.548854] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:36:52 (1786757812) [ 1956.003055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1957.399767] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:36:55 (1786757815) [ 1958.688781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1959.958651] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:36:58 (1786757818) [ 1960.174792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.199901] LNet: 112909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.207801] LNetError: 112909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.223882] LNet: Removed LNI 192.168.206.7@tcp [ 1960.253299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.305674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.351100] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1961.099667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1961.168268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1967.002476] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:37:04 (1786757824) [ 1967.131095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.168222] LNet: 113613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.189970] LNetError: 113613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.216199] LNet: Removed LNI 192.168.206.7@tcp [ 1967.270430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.347781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1967.385318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.111276] Key type .llcrypt unregistered [ 1968.116077] Key type ._llcrypt unregistered [ 1978.577613] Key type ._llcrypt registered [ 1978.579758] Key type .llcrypt registered [ 1979.611566] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:37:17 (1786757837) [ 1979.950332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1979.962529] alg: No test for adler32 (adler32-zlib) [ 1980.838767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.891599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.913911] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 1981.447431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1981.484699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 1986.988916] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:37:24 (1786757844) [ 1987.208551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.249129] LNet: 115177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.253365] LNetError: 115177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.280467] LNet: Removed LNI 192.168.206.7@tcp [ 1987.299522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.352426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1987.380877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1993.454237] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:37:30 (1786757850) [ 1993.636707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.712636] LNet: 115752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.757945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.914829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1994.521625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1994.549387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1994.606243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1994.989862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1995.044500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1995.130769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.148927] LNet: 115792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.166933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.216361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1995.821291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1995.891908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1995.948498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1996.530418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1996.583409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2002.741178] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:37:40 (1786757860) [ 2002.941579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.990889] LNet: 116392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.024577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.096721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.136891] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2003.977588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2004.017246] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2004.768871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2004.851602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2005.064099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.092768] LNetError: 116672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.112147] LNet: Removed LNI 192.168.206.7@tcp [ 2006.115443] LNetError: 116672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.136575] LNet: Removed LNI 192.168.206.7@tcp2 [ 2006.186334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.271071] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2006.279282] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2006.288959] LNet: 116679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2006.862753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2007.029432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.045586] LNet: 116716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.052891] LNet: 116716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2008.096135] LNet: Removed LNI 192.168.206.7@tcp [ 2008.108223] LNet: Removed LNI 192.168.206.7@tcp2 [ 2008.151228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.202632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2008.236907] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2008.967282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2009.006866] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2009.737874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2009.785434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2009.848419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2010.092691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.124338] LNetError: 117000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.129275] LNetError: 117000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2010.141217] LNet: Removed LNI 192.168.206.7@tcp [ 2011.176027] LNet: Removed LNI 192.168.206.7@tcp2 [ 2011.200760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.301069] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2011.313040] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2011.329802] LNet: 117006:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2011.834897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2012.048479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.073467] LNet: Removed LNI 192.168.206.7@tcp [ 2013.106708] LNet: Removed LNI 192.168.206.7@tcp2 [ 2013.138208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.245710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.287576] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2014.216851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2014.250071] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2015.166722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2015.235287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2015.611762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.656603] LNet: 117323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.668532] LNet: 117323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2015.686670] LNetError: 117323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.697207] LNetError: 117323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2015.709190] LNet: Removed LNI 192.168.206.7@tcp [ 2015.725645] LNet: Removed LNI 192.168.206.7@tcp2 [ 2015.757827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.817523] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2015.821176] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2015.825184] LNet: 117329:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2016.578957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2016.836276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.888133] LNet: Removed LNI 192.168.206.7@tcp [ 2017.910544] LNet: Removed LNI 192.168.206.7@tcp2 [ 2017.958353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.076873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2018.116052] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2018.981040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2019.007757] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2020.010912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2020.103130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2020.208649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2020.481432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.536152] LNet: Removed LNI 192.168.206.7@tcp [ 2022.567307] LNet: Removed LNI 192.168.206.7@tcp2 [ 2022.635436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.744141] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2022.755039] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2022.766514] LNet: 117657:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2023.577534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2023.781239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2023.798350] LNetError: 117694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.805273] LNetError: 117694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2023.812493] LNet: Removed LNI 192.168.206.7@tcp [ 2024.892203] LNet: Removed LNI 192.168.206.7@tcp2 [ 2024.937034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.992385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.032818] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2025.841714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2025.868444] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2026.652434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2026.703547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2026.766984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2027.045480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.128382] LNet: Removed LNI 192.168.206.7@tcp [ 2028.150497] LNet: Removed LNI 192.168.206.7@tcp2 [ 2028.224419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.344757] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2028.349814] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2028.355089] LNet: 117997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2028.875415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2029.277206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.368566] LNet: Removed LNI 192.168.206.7@tcp [ 2030.385831] LNet: Removed LNI 192.168.206.7@tcp2 [ 2030.405863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.491399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.531753] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2031.214078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2031.238863] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2032.085705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2032.148895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2032.180936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2032.228435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2032.537049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.557455] LNet: 118350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.560699] LNet: 118350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2033.632300] LNet: Removed LNI 192.168.206.7@tcp [ 2034.662779] LNet: Removed LNI 192.168.206.7@tcp2 [ 2034.712044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.794508] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2034.803540] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2035.371056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2041.398532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2043.238957] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:38:20 (1786757900) [ 2043.353871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.393664] LNetError: 119111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.408666] LNetError: 119111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2043.421095] LNet: Removed LNI 192.168.206.7@tcp [ 2044.459059] LNet: Removed LNI 192.168.206.7@tcp2 [ 2044.503491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.799693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2044.825109] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2045.495565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2045.513239] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2046.249798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.291461] LNet: Removed LNI 10.1.2.3@tcp [ 2046.304441] LNet: Removed LNI 192.168.206.7@tcp [ 2046.324775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.390192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2046.406423] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2046.409589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2046.896485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2046.913233] LNet: Removed LNI 192.168.206.7@tcp [ 2046.927506] LNet: Removed LNI 10.1.2.3@tcp [ 2053.047672] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:38:30 (1786757910) [ 2053.137373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.185421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.244794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.264622] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2054.064284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2054.104309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.178463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.240316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.289185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.330641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.368392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.433375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.471265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.499058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.531381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.7@tcp [ 2054.689683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2060.577581] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:38:38 (1786757918) [ 2060.647325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.735373] LNet: Removed LNI 192.168.206.7@tcp [ 2061.782916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.903056] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2061.976377] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2061.979967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2061.986949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2064.468218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.496856] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2065.213871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2065.231217] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2067.107967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2067.173484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2067.221834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.7@tcp -m PUT -e local_timeout -r 1 [ 2067.269487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2067.315437] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2067.322952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2067.522823] LNetError: 120797:0:(lib-move.c:4801:lnet_parse()) 192.168.206.7@tcp, src 192.168.206.7@tcp: Dropping PUT (error -108 looking up sender) [ 2068.579808] LNet: 121259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.584562] LNet: 121259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2069.600313] LNet: Removed LNI 10.1.2.3@tcp [ 2070.631891] LNet: Removed LNI 192.168.206.7@tcp [ 2071.021184] Key type .llcrypt unregistered [ 2071.024402] Key type ._llcrypt unregistered [ 2081.405174] Key type ._llcrypt registered [ 2081.407576] Key type .llcrypt registered [ 2082.591516] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:39:00 (1786757940) [ 2083.147803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.162341] alg: No test for adler32 (adler32-zlib) [ 2084.102548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.460808] LNet: 122111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.084355] Key type .llcrypt unregistered [ 2085.088235] Key type ._llcrypt unregistered [ 2090.979545] Key type ._llcrypt registered [ 2090.983881] Key type .llcrypt registered [ 2091.210601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2091.222895] alg: No test for adler32 (adler32-zlib) [ 2092.106166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2092.133032] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2098.987057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 2099.198127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2099.800718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.107@tcp [ 2100.021972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2100.044130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2100.067603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2100.094150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2100.121444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2100.147562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2100.170602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2100.217566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2100.259091] LNet: 122563:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2100.296633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2100.321246] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.107@tcp [ 2100.329858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2100.396981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2100.429170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2100.470807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2100.511613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2101.051905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2101.274264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2101.297828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2101.326248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2101.353935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2101.380788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2101.409796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2101.432288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2101.454753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2101.471612] LNet: 122680:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2101.493115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2101.535434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2101.572985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2101.597458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2101.634643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2101.659245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2102.395667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.107@tcp [ 2102.742968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2102.782273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2102.808996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2102.843591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2102.873415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2102.901995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2102.958180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2102.989971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2103.009371] LNet: 122798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.028539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2103.054659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2103.098893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2103.140859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2103.166935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2103.210115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2103.897808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2104.138679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2104.172376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2104.202173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2104.250796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2104.290544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2104.333830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2104.398942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2104.446586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2104.491953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2104.527958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2104.570295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2104.605512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2104.633601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2104.669586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2105.014499] LNet: 122965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.019474] LNetError: 122965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.036529] LNet: Removed LNI 192.168.206.7@tcp [ 2105.416794] Key type .llcrypt unregistered [ 2105.419360] Key type ._llcrypt unregistered [ 2123.480755] Key type ._llcrypt registered [ 2123.490779] Key type .llcrypt registered [ 2124.759826] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:39:42 (1786757982) [ 2125.067462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.087809] alg: No test for adler32 (adler32-zlib) [ 2126.048901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.376931] LNet: 123852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.276543] Key type .llcrypt unregistered [ 2127.283555] Key type ._llcrypt unregistered [ 2133.431536] Key type ._llcrypt registered [ 2133.435543] Key type .llcrypt registered [ 2133.714692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2133.735344] alg: No test for adler32 (adler32-zlib) [ 2134.641583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2134.661896] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2141.597525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.107@tcp [ 2150.008207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.046876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2150.069066] LNet: Added LNI 192.168.206.7@tcp1 [8/256/0/180] [ 2150.807376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2151.331816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.107@tcp [ 2151.564569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2151.600928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2151.631381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2151.657948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2151.705834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2151.746427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2151.778197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2151.818667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2151.846522] LNet: 124508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.869223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2151.904600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2151.934621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2151.958286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2151.988524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2152.010988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2152.562920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2152.810347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2152.840374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2152.874753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2152.894607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2152.914683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2152.934668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2152.955154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2152.978331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2152.993336] LNet: 124625:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2153.010540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2153.029451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2153.058316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2153.097388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2153.120451] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.107@tcp, 192.168.206.107@tcp1 [ 2153.135091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2153.174067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2153.838421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.107@tcp [ 2154.014394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2154.043761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2154.073276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2154.098250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2154.135716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2154.173089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2154.206871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2154.250412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2154.274989] LNet: 124742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.309742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2154.332352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2154.355611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2154.381533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2154.408598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2154.432932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2154.951039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2155.152423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2155.184992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2155.210290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2155.257517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2155.299329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2155.349227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2155.378841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2155.433050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2155.479086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2155.524301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2155.553376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2155.600085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2155.628891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2155.650181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2160.074315] LNet: 124944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.083657] LNetError: 124944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.090613] LNet: Removed LNI 192.168.206.7@tcp [ 2160.104829] LNet: Removed LNI 192.168.206.7@tcp1 [ 2160.425260] Key type .llcrypt unregistered [ 2160.429103] Key type ._llcrypt unregistered [ 2174.915822] Key type ._llcrypt registered [ 2174.918212] Key type .llcrypt registered [ 2176.194778] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:40:33 (1786758033) [ 2176.646280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.662848] alg: No test for adler32 (adler32-zlib) [ 2177.571600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.624961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2177.648136] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2178.259595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2178.275386] LNet: Added LNI 192.168.206.7@tcp2 [8/256/0/180] [ 2179.041285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.7@tcp [ 2179.106194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2180.192557] LNet: Removed LNI 192.168.206.7@tcp [ 2185.435543] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:40:43 (1786758043) [ 2186.660091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2188.335224] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:40:46 (1786758046) [ 2195.075784] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:40:52 (1786758052) [ 2200.605891] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:40:58 (1786758058) [ 2206.859692] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:41:04 (1786758064) [ 2213.078478] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:41:11 (1786758071) [ 2220.065668] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:41:17 (1786758077) [ 2226.177845] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:41:24 (1786758084) [ 2232.969945] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:41:30 (1786758090) [ 2240.260968] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:41:37 (1786758097) [ 2246.216906] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:41:44 (1786758104) [ 2252.098351] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:41:50 (1786758110) [ 2258.354694] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:41:56 (1786758116) [ 2264.488522] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:42:02 (1786758122) [ 2271.247955] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:42:09 (1786758129) [ 2278.567564] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:42:16 (1786758136) [ 2285.729463] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:42:23 (1786758143) [ 2293.159966] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:42:30 (1786758150) [ 2300.597236] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:42:38 (1786758158) [ 2300.851804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.875608] LNet: 136634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.885789] LNetError: 136634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.901069] LNet: Removed LNI 192.168.206.7@tcp2 [ 2300.941792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.076341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.102508] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2302.178914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.220546] LNet: 136776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.229501] LNetError: 136776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.242875] LNet: Removed LNI 192.168.206.7@tcp [ 2302.257180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.305513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2302.322755] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2308.312837] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:42:46 (1786758166) [ 2314.659177] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:42:52 (1786758172) [ 2322.103859] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:42:59 (1786758179) [ 2328.747544] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:43:06 (1786758186) [ 2335.747409] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:43:13 (1786758193) [ 2342.503363] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:43:20 (1786758200) [ 2350.154592] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:43:28 (1786758208) [ 2356.887595] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:43:34 (1786758214) [ 2363.241710] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:43:41 (1786758221) [ 2369.697886] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:43:47 (1786758227) [ 2375.763650] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:43:53 (1786758233) [ 2383.189324] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:44:00 (1786758240) [ 2390.605833] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2151 sec ========== 21:44:08 (1786758248) [ 2390.945535] LNet: 144244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.962651] LNetError: 144244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.998233] LNet: Removed LNI 192.168.206.7@tcp [ 2391.775659] Key type .llcrypt unregistered [ 2391.780082] Key type ._llcrypt unregistered [ 2404.742315] Key type ._llcrypt registered [ 2404.746133] Key type .llcrypt registered [ 2405.121801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2405.139206] alg: No test for adler32 (adler32-zlib) [ 2406.242900] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2406.433107] LNet: Added LNI 192.168.206.7@tcp [8/256/0/180] [ 2408.144585] Key type lgssc registered [ 2409.316883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2479.336232] Lustre: Mounted lustre-client - version 2.17.57_1_g2c756e6 [ 2485.181404] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2494.749882] Lustre: Unmounted lustre-client [ 2525.963429] Key type lgssc unregistered [ 2526.225162] LNet: 147204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.230292] LNetError: 147204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.267131] LNet: Removed LNI 192.168.206.7@tcp [ 2527.218283] Key type .llcrypt unregistered [ 2527.223588] Key type ._llcrypt unregistered