[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 395685455 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002231] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006617] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007011] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008055] pid_max: default: 32768 minimum: 301 [ 0.009244] LSM: Security Framework initializing [ 0.010168] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.012134] *** VALIDATE selinux *** [ 0.023671] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027550] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028094] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030076] *** VALIDATE tmpfs *** [ 0.032070] *** VALIDATE proc *** [ 0.032897] *** VALIDATE cgroup *** [ 0.033004] *** VALIDATE cgroup2 *** [ 0.034169] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035108] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036020] Spectre V2 : User space: Vulnerable [ 0.037004] Speculative Store Bypass: Vulnerable [ 0.040219] debug: unmapping init [mem 0xffffffffb4059000-0xffffffffb4060fff] [ 0.042643] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043400] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044010] ... version: 2 [ 0.044931] ... bit width: 48 [ 0.045006] ... generic registers: 4 [ 0.045840] ... value mask: 0000ffffffffffff [ 0.046005] ... max period: 00007fffffffffff [ 0.047005] ... fixed-purpose events: 3 [ 0.047897] ... event mask: 000000070000000f [ 0.049230] rcu: Hierarchical SRCU implementation. [ 0.051160] smp: Bringing up secondary CPUs ... [ 0.052499] x86: Booting SMP configuration: [ 0.053014] .... node #0, CPUs: #1 #2 #3 [ 0.055393] smp: Brought up 1 node, 4 CPUs [ 0.057008] smpboot: Max logical packages: 1 [ 0.058010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.267028] node 0 deferred pages initialised in 208ms [ 0.270273] devtmpfs: initialized [ 0.271216] x86/mm: Memory block size: 128MB [ 0.273150] gcov: version magic: 0x41383552 [ 0.276208] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.279076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.282283] pinctrl core: initialized pinctrl subsystem [ 0.284151] [ 0.284737] ************************************************************* [ 0.287015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289009] ** ** [ 0.291013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.293010] ** ** [ 0.295011] ** This means that this kernel is built to expose internal ** [ 0.296006] ** IOMMU data structures, which may compromise security on ** [ 0.298007] ** your system. ** [ 0.300008] ** ** [ 0.301006] ** If you see this message and you are not debugging the ** [ 0.302006] ** kernel, report this immediately to your vendor! ** [ 0.304005] ** ** [ 0.305005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.307010] ************************************************************* [ 0.308530] NET: Registered protocol family 16 [ 0.310398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.312040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.314029] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316525] cpuidle: using governor menu [ 0.317389] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.320419] PCI: Using configuration type 1 for base access [ 0.322150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.329120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.330010] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331154] cryptd: max_cpu_qlen set to 1000 [ 0.332243] ACPI: Added _OSI(Module Device) [ 0.334015] ACPI: Added _OSI(Processor Device) [ 0.335009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.337066] ACPI: Added _OSI(Processor Aggregator Device) [ 0.340912] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.346566] ACPI: Interpreter enabled [ 0.348058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349007] ACPI: Using IOAPIC for interrupt routing [ 0.351117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.369015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376327] acpiphp: Slot [2] registered [ 0.377078] acpiphp: Slot [5] registered [ 0.379080] acpiphp: Slot [6] registered [ 0.380075] acpiphp: Slot [3] registered [ 0.381056] acpiphp: Slot [4] registered [ 0.382074] acpiphp: Slot [7] registered [ 0.383067] acpiphp: Slot [8] registered [ 0.384077] acpiphp: Slot [9] registered [ 0.386056] acpiphp: Slot [10] registered [ 0.387057] acpiphp: Slot [11] registered [ 0.388058] acpiphp: Slot [12] registered [ 0.389072] acpiphp: Slot [13] registered [ 0.391062] acpiphp: Slot [14] registered [ 0.392059] acpiphp: Slot [15] registered [ 0.393055] acpiphp: Slot [16] registered [ 0.394059] acpiphp: Slot [17] registered [ 0.395072] acpiphp: Slot [18] registered [ 0.396061] acpiphp: Slot [19] registered [ 0.398104] acpiphp: Slot [20] registered [ 0.399060] acpiphp: Slot [21] registered [ 0.400057] acpiphp: Slot [22] registered [ 0.401070] acpiphp: Slot [23] registered [ 0.402058] acpiphp: Slot [24] registered [ 0.404099] acpiphp: Slot [25] registered [ 0.405067] acpiphp: Slot [26] registered [ 0.406101] acpiphp: Slot [27] registered [ 0.407102] acpiphp: Slot [28] registered [ 0.408113] acpiphp: Slot [29] registered [ 0.410077] acpiphp: Slot [30] registered [ 0.411061] acpiphp: Slot [31] registered [ 0.413062] PCI host bridge to bus 0000:00 [ 0.414012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.419013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.421013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.424025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.426017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.427159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.429878] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.432975] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.438418] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.441465] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.445009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.447008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.451665] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.453030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.456778] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.459998] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.469013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.473010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.477766] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.481863] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.485010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.496872] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.506398] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.510789] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514718] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.526017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.534588] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.536285] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.538212] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.540269] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.542254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.546226] iommu: Default domain type: Passthrough [ 0.548455] SCSI subsystem initialized [ 0.549114] ACPI: bus type USB registered [ 0.551090] usbcore: registered new interface driver usbfs [ 0.553078] usbcore: registered new interface driver hub [ 0.555061] usbcore: registered new device driver usb [ 0.556135] pps_core: LinuxPPS API ver. 1 registered [ 0.558009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.560076] PTP clock support registered [ 0.562115] EDAC MC: Ver: 3.0.0 [ 0.563533] PCI: Using ACPI for IRQ routing [ 0.565725] NetLabel: Initializing [ 0.566007] NetLabel: domain hash size = 128 [ 0.567006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.568045] NetLabel: unlabeled traffic allowed by default [ 0.569089] vgaarb: loaded [ 0.570234] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.571008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.576009] clocksource: Switched to clocksource kvm-clock [ 0.653211] VFS: Disk quotas dquot_6.6.0 [ 0.654140] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.655686] *** VALIDATE ramfs *** [ 0.656542] *** VALIDATE hugetlbfs *** [ 0.658252] pnp: PnP ACPI init [ 0.660381] pnp: PnP ACPI: found 6 devices [ 0.672862] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.675168] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.676389] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.678234] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.679614] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.681104] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.683345] NET: Registered protocol family 2 [ 0.685190] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.688390] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.690285] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.693858] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.696086] TCP: Hash tables configured (established 65536 bind 65536) [ 0.697823] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.699837] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.701990] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.704129] NET: Registered protocol family 1 [ 0.706281] RPC: Registered named UNIX socket transport module. [ 0.707718] RPC: Registered udp transport module. [ 0.708740] RPC: Registered tcp transport module. [ 0.709658] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.711178] NET: Registered protocol family 44 [ 0.712870] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.714270] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.715580] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.717839] PCI: CLS 0 bytes, default 64 [ 0.719059] Unpacking initramfs... [ 1.954827] debug: unmapping init [mem 0xffff9821bcc64000-0xffff9821bffcffff] [ 1.958071] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.959742] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.961960] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.565679] Initialise system trusted keyrings [ 2.567472] Key type blacklist registered [ 2.569184] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.579666] zbud: loaded [ 2.582454] *** VALIDATE nfs *** [ 2.583599] *** VALIDATE nfs4 *** [ 2.585264] pstore: using deflate compression [ 2.589083] Platform Keyring initialized [ 2.696950] NET: Registered protocol family 38 [ 2.698974] Key type asymmetric registered [ 2.700839] Asymmetric key parser 'x509' registered [ 2.702994] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.706893] io scheduler mq-deadline registered [ 2.709469] io scheduler kyber registered [ 2.713427] io scheduler bfq registered [ 2.718447] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.723054] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.727824] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.730728] ACPI: Power Button [PWRF] [ 2.735829] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.742123] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.773929] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.806589] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.835904] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.842444] Non-volatile memory driver v1.3 [ 2.844132] Linux agpgart interface v0.103 [ 2.871857] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.873909] vda: detected capacity change from 0 to 74706944 [ 2.888525] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.891260] vdb: detected capacity change from 0 to 1073741824 [ 2.896596] libphy: Fixed MDIO Bus: probed [ 2.905455] usbcore: registered new interface driver usbserial_generic [ 2.907701] usbserial: USB Serial support registered for generic [ 2.909550] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.913287] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.915933] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.918937] mousedev: PS/2 mouse device common for all mice [ 2.922537] rtc_cmos 00:05: RTC can wake from S4 [ 2.924637] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.927916] rtc_cmos 00:05: registered as rtc0 [ 2.933259] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.934041] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.941063] intel_pstate: CPU model not supported [ 2.943200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.947574] hid: raw HID events driver (C) Jiri Kosina [ 2.949659] usbcore: registered new interface driver usbhid [ 2.952362] usbhid: USB HID core driver [ 2.954353] drop_monitor: Initializing network drop monitor service [ 2.956860] Initializing XFRM netlink socket [ 2.958702] NET: Registered protocol family 10 [ 2.961552] Segment Routing with IPv6 [ 2.962833] NET: Registered protocol family 17 [ 2.964716] mpls_gso: MPLS GSO support [ 2.969839] RAS: Correctable Errors collector initialized. [ 2.971603] AVX version of gcm_enc/dec engaged. [ 2.972906] AES CTR mode by8 optimization enabled [ 3.046532] sched_clock: Marking stable (3046515963, 0)->(3787200751, -740684788) [ 3.054562] registered taskstats version 1 [ 3.058916] Loading compiled-in X.509 certificates [ 3.062629] zswap: loaded using pool lzo/zbud [ 3.104227] Key type big_key registered [ 3.116763] Key type encrypted registered [ 3.118338] ima: No TPM chip found, activating TPM-bypass! [ 3.124878] ima: Allocated hash algorithm: sha1 [ 3.128383] ima: No architecture policies found [ 3.131883] evm: Initialising EVM extended attributes: [ 3.138091] evm: security.selinux [ 3.139191] evm: security.ima [ 3.142742] evm: security.capability [ 3.145297] evm: HMAC attrs: 0x1 [ 3.147922] rtc_cmos 00:05: setting system clock to 2026-08-17 02:38:26 UTC (1786934306) [ 3.155103] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 3.158336] debug: unmapping init [mem 0xffffffffb3d82000-0xffffffffb4058fff] [ 3.166358] Write protecting the kernel read-only data: 28672k [ 3.170304] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 3.173765] debug: unmapping init [mem 0xffffffffb2d14000-0xffffffffb2dfffff] [ 3.214898] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.226114] systemd[1]: Detected virtualization kvm. [ 3.232340] systemd[1]: Detected architecture x86-64. [ 3.234802] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.279759] systemd[1]: No hostname configured. [ 3.283463] systemd[1]: Set hostname to . [ 3.285608] random: systemd: uninitialized urandom read (16 bytes read) [ 3.288932] systemd[1]: Initializing machine ID from random generator. [ 3.495538] random: ln: uninitialized urandom read (6 bytes read) [ 3.629148] random: systemd: uninitialized urandom read (16 bytes read) [ 3.635964] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.649935] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.660782] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.967488] device-mapper: uevent: version 1.0.3 [ 4.973181] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 6.444583] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.862213] virtio_net virtio0 ens2: renamed from eth0 [ 8.745707] scsi host0: ata_piix [ 8.913495] scsi host1: ata_piix [ 8.921429] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.927814] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.668139] random: crng init done [ 12.670335] random: 7 urandom warning(s) missed due to ratelimiting [ 15.141492] dracut-initqueue[598]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.876300] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.240554] printk: systemd: 21 output lines suppressed due to ratelimiting [ 19.764637] SELinux: Disabled at runtime. [ 19.829170] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.839615] systemd[1]: Detected virtualization kvm. [ 19.842840] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.362657] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.366649] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.374163] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.379449] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.383254] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.390964] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.397754] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [[ 21.902463] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.502038] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.614571] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.810680] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.132941] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.172763] EDAC sbridge: Ver: 1.1.2 [ 26.648482] Key type dns_resolver registered [ 27.158295] NFS: Registering the id_resolver key type [ 27.161452] Key type id_resolver registered [ 27.164569] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg311-client login: [ 68.001847] libcfs: loading out-of-tree module taints kernel. [ 68.066177] Key type ._llcrypt registered [ 68.070175] Key type .llcrypt registered [ 68.276124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 68.281190] alg: No test for adler32 (adler32-zlib) [ 69.260780] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 69.545467] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 71.144101] Key type lgssc registered [ 71.689808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.190457] Lustre: Mounted lustre-client - version 2.17.57_1_g37263e8 [ 122.495471] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 131.297726] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing check_logdir /tmp/testlogs/ [ 132.857629] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing yml_node [ 134.471728] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 135.374952] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 136.245561] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 136.805736] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Aug 16 22:40:39 EDT 2026 [ 140.768185] Lustre: lustre-OST0000-osc-ffff98221872e000: disconnect after 20s idle [ 141.486499] Lustre: Unmounted lustre-client [ 149.840674] Key type lgssc unregistered [ 149.962209] LNet: 8427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 149.967771] LNetError: 8427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 149.978556] LNet: Removed LNI 192.168.203.11@tcp [ 150.288121] Key type .llcrypt unregistered [ 150.290052] Key type ._llcrypt unregistered [ 159.134939] Key type ._llcrypt registered [ 159.136368] Key type .llcrypt registered [ 159.207656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 159.216460] alg: No test for adler32 (adler32-zlib) [ 160.024994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 160.053211] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 160.062076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 161.557485] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 161.590051] LNet: 9305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 161.594838] LNetError: 9305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 161.603598] LNet: Removed LNI 192.168.203.11@tcp [ 161.870609] Key type .llcrypt unregistered [ 161.872665] Key type ._llcrypt unregistered [ 164.030598] Key type ._llcrypt registered [ 164.032028] Key type .llcrypt registered [ 164.550880] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:41:07 (1786934467) [ 164.684489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 164.691644] alg: No test for adler32 (adler32-zlib) [ 165.493685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.525627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.536562] LNet: 9725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 165.546363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.570493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 165.579608] LNet: 9732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 168.058420] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:41:10 (1786934470) [ 168.108142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.119139] LNet: 10308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 168.128882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.161387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 170.716332] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:41:13 (1786934473) [ 170.769648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.782234] LNet: 10892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 170.794079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.825817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 173.554623] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:41:16 (1786934476) [ 173.597812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.609482] LNet: 11472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.622125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.660836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 176.154158] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:41:18 (1786934478) [ 176.207774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.231334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.273119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 176.472335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 179.209798] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:41:22 (1786934482) [ 179.253556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.265554] LNet: 12644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 179.268895] LNet: 12644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 179.283489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.323490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 179.482871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 179.648505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 182.126027] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:41:24 (1786934484) [ 182.157359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 182.174587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 182.214434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 184.728986] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:41:27 (1786934487) [ 184.768987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.790258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.824727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 184.847175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 185.011962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 185.028884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 185.195858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 185.213192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 185.369084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 185.386539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 185.552851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 185.569935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 185.727846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 185.743469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 185.898240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 185.917788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 186.073547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 186.089033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 186.258103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 186.275394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 186.454888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 186.472827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 186.628150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 186.652688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 186.824825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 186.858683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 187.030613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 187.049963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 189.869716] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:41:32 (1786934492) [ 189.918437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.929298] LNet: 14505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 189.932607] LNet: 14505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 189.945328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.983790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 190.003781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 193.010886] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:41:35 (1786934495) [ 193.067547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.090316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.131602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 193.152454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 195.866252] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:41:38 (1786934498) [ 195.967073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.992499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.033690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 196.052488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 198.904245] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:41:41 (1786934501) [ 198.942284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.958261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.003248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 199.023755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 201.867114] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:41:44 (1786934504) [ 201.907425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.927173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.967320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 201.985198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 204.823279] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:41:47 (1786934507) [ 204.869652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.887933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.935184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 204.957081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 207.766697] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:41:50 (1786934510) [ 207.815088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.828582] LNet: 18018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 207.832383] LNet: 18018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 207.847500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.898892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 208.079613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 210.934376] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:41:53 (1786934513) [ 210.991180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.013854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.118245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 211.223944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 214.097975] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:41:56 (1786934516) [ 214.193820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.210094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.262345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 214.380454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 217.207413] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:42:00 (1786934520) [ 217.284775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.302226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.532122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 217.640754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 220.222691] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:42:03 (1786934523) [ 220.272972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.292511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.335341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 220.363261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 223.052406] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:42:05 (1786934525) [ 223.115450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.141314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.179543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 223.210571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 226.004709] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:42:08 (1786934528) [ 226.034129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.056252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.093269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 228.665906] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:42:11 (1786934531) [ 228.705686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 228.729477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 228.766419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 231.246181] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:42:14 (1786934534) [ 231.282068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.307685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.347346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 231.511743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 233.923512] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:42:16 (1786934536) [ 233.956972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.974622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.015245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 234.030411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 236.563413] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:42:19 (1786934539) [ 236.620648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.637921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.670532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 236.686570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 239.176083] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:42:21 (1786934541) [ 239.234785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.252472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.283852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 239.302821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 241.849550] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:42:24 (1786934544) [ 241.906585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.921647] LNet: 25213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 241.925096] LNet: 25213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 241.935839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.968589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 241.989190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 244.504620] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:42:27 (1786934547) [ 244.588380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.615480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.640418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 244.675739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 244.687497] LNetError: 25812:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 244.708911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 244.735163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 247.239216] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:42:30 (1786934550) [ 247.280935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.298694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.340722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BqBMSRR6hp [ 247.374990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BqBMSRR6hp [ 247.399857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BqBMSRR6hp [ 249.860264] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:42:32 (1786934552) [ 249.912735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.931714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.954239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 249.969656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 249.990034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 252.290319] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:42:35 (1786934555) [ 252.322612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.346375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.389111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 252.400600] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 253.032930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 253.534247] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 255.232166] Key type lgssc registered [ 255.723524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 257.300255] LNetError: 27569:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.11@tcp99 has no available nets [ 258.485718] Key type lgssc unregistered [ 258.664768] LNetError: 28681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 258.675316] LNet: Removed LNI 192.168.203.11@tcp [ 259.003273] Key type .llcrypt unregistered [ 259.004791] Key type ._llcrypt unregistered [ 261.418151] Key type ._llcrypt registered [ 261.419768] Key type .llcrypt registered [ 262.006813] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:42:44 (1786934564) [ 262.144737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 262.153071] alg: No test for adler32 (adler32-zlib) [ 262.983742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.012617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 264.147276] LNet: Message forwarding will be enabled in 2s [ 267.232159] LNet: Message forwarding enabled [ 267.348124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 267.358861] LNet: Message forwarding will stop in 2s to 6s [ 271.540518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 272.035946] LNet: Message forwarding disabled [ 272.560435] LNet: Message forwarding will be enabled in 2s [ 275.566213] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:42:58 (1786934578) [ 275.644057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.654232] LNet: 30087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.223469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.258728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 278.952569] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:43:01 (1786934581) [ 278.992286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.004825] LNet: 30661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.016723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.047663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 279.066083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 279.083861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 279.096193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 279.109341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 279.132403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 279.152296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 279.185559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 279.206584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 279.230893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 279.247538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 279.263325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 279.280855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 279.299594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 279.320716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 279.343413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 279.363353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 279.382781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 279.403315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 279.418746] LNetError: 30711:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 282.071301] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:43:04 (1786934584) [ 282.107437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.118142] LNet: 31277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.129666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.164770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 284.964234] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:43:07 (1786934587) [ 285.752529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg311-server [ 286.460910] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:43:09 (1786934589) [ 286.538200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.547861] LNet: 32211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.566027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.590881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 286.602226] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 286.972914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[12-16]@tcp [ 286.986766] LNet: 32344:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 287.008505] LNetError: 32215:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 287.017640] LNetError: 32215:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.12@tcp has gone from up to down [ 288.039907] LNetError: 32215:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 288.046988] LNetError: 32215:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 289.060074] LNetError: 32215:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 289.677167] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:43:12 (1786934592) [ 290.080758] LNetError: 32219:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.16@tcp has gone from up to down [ 290.573248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg311-server [ 291.114026] LNetError: 32215:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 291.120195] LNetError: 32215:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 291.320877] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:43:14 (1786934594) [ 291.378787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.130152] LNetError: 32214:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.203.12@tcp [ 293.473594] LNet: 33277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.479454] LNetError: 33277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.492111] LNet: Removed LNI 192.168.203.11@tcp [ 293.505142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.541809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 293.554358] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 293.880458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[12-17/2]@tcp [ 293.891822] LNet: 33396:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 293.896712] LNet: 33396:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 293.906946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[12-17/2]@tcp [ 293.911705] LNetError: 33283:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.12@tcp has gone from up to down [ 293.922904] LNetError: 33283:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 296.567760] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:43:19 (1786934599) [ 296.619662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.629338] LNetError: 33985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.637274] LNet: Removed LNI 192.168.203.11@tcp [ 296.650564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.686937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 296.713378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.728686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.763702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.793408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.810610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.835877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 296.874598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.896429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.932414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.974400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.997419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.030580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 297.075597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.099857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.133513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.171708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.192750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.219986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 297.256770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.275692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.309616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.348063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.372658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.401611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 297.439570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.457663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.490952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.524971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.544621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.577288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 297.605542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.624739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.651856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.334704] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:43:23 (1786934603) [ 301.181614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg311-server [ 301.945310] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:43:24 (1786934604) [ 302.775551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg311-server [ 303.403800] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:43:26 (1786934606) [ 303.442435] LNet: 35379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.444768] LNet: 35379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 303.715505] Key type .llcrypt unregistered [ 303.718148] Key type ._llcrypt unregistered [ 305.992414] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 306.046383] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 306.048841] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 306.051991] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 308.308321] Key type ._llcrypt registered [ 308.309488] Key type .llcrypt registered [ 308.395199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 308.403306] alg: No test for adler32 (adler32-zlib) [ 309.219393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.243840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 309.255039] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 309.532707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 309.544221] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 310.357269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 310.368380] LNet: Removed LNI 10.1.2.3@tcp [ 310.415951] LNet: 36107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.420402] LNetError: 36107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.428933] LNet: Removed LNI 192.168.203.11@tcp [ 310.704520] Key type .llcrypt unregistered [ 310.705943] Key type ._llcrypt unregistered [ 314.995408] Key type ._llcrypt registered [ 314.996924] Key type .llcrypt registered [ 315.549492] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:43:38 (1786934618) [ 315.750249] Key type .llcrypt unregistered [ 315.751536] Key type ._llcrypt unregistered [ 317.970323] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 318.041918] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 318.044052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 318.046243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 320.267253] Key type ._llcrypt registered [ 320.268916] Key type .llcrypt registered [ 320.351181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 320.357974] alg: No test for adler32 (adler32-zlib) [ 321.178904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.208429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 321.220089] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 321.509499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 321.519044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 322.524878] LNet: 37605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.528514] LNetError: 37605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 322.531896] LNet: Removed LNI 10.1.2.3@tcp [ 322.540447] LNet: Removed LNI 192.168.203.11@tcp [ 322.828488] Key type .llcrypt unregistered [ 322.830601] Key type ._llcrypt unregistered [ 326.867801] Key type ._llcrypt registered [ 326.869325] Key type .llcrypt registered [ 327.366533] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:43:50 (1786934630) [ 327.571405] Key type .llcrypt unregistered [ 327.573210] Key type ._llcrypt unregistered [ 329.755615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 329.795743] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 329.798306] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 329.801540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 332.082768] Key type ._llcrypt registered [ 332.083902] Key type .llcrypt registered [ 332.160394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 332.167937] alg: No test for adler32 (adler32-zlib) [ 332.987347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.013514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 333.026987] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 333.868784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 333.882847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 334.773463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 334.783527] LNet: Removed LNI 10.1.2.3@tcp [ 334.790931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 334.813370] LNet: Removed LNI 10.1.2.31@tcp [ 334.822439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 334.834149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 335.160906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 335.174969] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 335.637572] LNet: 39530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.641412] LNetError: 39530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 335.646825] LNet: Removed LNI 10.1.2.31@tcp [ 335.655871] LNet: Removed LNI 10.1.2.3@tcp [ 335.967558] Key type .llcrypt unregistered [ 335.971212] Key type ._llcrypt unregistered [ 340.490837] Key type ._llcrypt registered [ 340.492500] Key type .llcrypt registered [ 341.011555] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:44:03 (1786934643) [ 341.152536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 341.160068] alg: No test for adler32 (adler32-zlib) [ 341.995972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.026639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.040720] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 342.373248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.384798] LNet: 40487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.388573] LNetError: 40487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 342.396389] LNet: Removed LNI 192.168.203.11@tcp [ 342.405922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.433667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.11@tcp [ 342.447589] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 342.771403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.11@tcp [ 342.792316] LNet: Removed LNI 192.168.203.11@tcp [ 343.083353] Key type .llcrypt unregistered [ 343.084693] Key type ._llcrypt unregistered [ 347.287718] Key type ._llcrypt registered [ 347.289262] Key type .llcrypt registered [ 347.785226] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:44:10 (1786934650) [ 348.465164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg311-server [ 349.106662] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:44:11 (1786934651) [ 349.318270] Key type .llcrypt unregistered [ 349.320041] Key type ._llcrypt unregistered [ 351.647274] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 351.718162] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 351.719875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 351.722281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 353.971732] Key type ._llcrypt registered [ 353.973057] Key type .llcrypt registered [ 354.046266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 354.054323] alg: No test for adler32 (adler32-zlib) [ 354.872371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.904320] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 355.220811] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 356.103413] LNet: 42463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.109909] LNetError: 42463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.115885] LNet: Removed LNI 192.168.203.11@tcp1 [ 356.126865] LNet: Removed LNI 10.1.2.3@tcp2 [ 356.404416] Key type .llcrypt unregistered [ 356.406340] Key type ._llcrypt unregistered [ 360.573762] Key type ._llcrypt registered [ 360.575829] Key type .llcrypt registered [ 361.151752] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:44:23 (1786934663) [ 361.303404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 361.310441] alg: No test for adler32 (adler32-zlib) [ 362.119260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.142524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 362.153951] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 362.522119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.533109] LNet: 43436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.536396] LNetError: 43436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.543404] LNet: Removed LNI 192.168.203.11@tcp [ 362.552130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.577968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 362.590399] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 365.266624] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:44:28 (1786934668) [ 365.304241] LNet: 44018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.308478] LNetError: 44018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 365.316622] LNet: Removed LNI 192.168.203.11@tcp [ 365.599870] Key type .llcrypt unregistered [ 365.601347] Key type ._llcrypt unregistered [ 367.971671] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 368.085480] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 368.087675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 368.091097] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 370.305995] Key type ._llcrypt registered [ 370.307793] Key type .llcrypt registered [ 370.378427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 370.384245] alg: No test for adler32 (adler32-zlib) [ 371.202638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.230160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 371.244281] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 371.599088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 371.609567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 372.601269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.610916] LNet: 44770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.614614] LNetError: 44770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 372.617842] LNet: Removed LNI 10.1.2.3@tcp [ 372.626797] LNet: Removed LNI 192.168.203.11@tcp [ 372.637246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.666515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 372.681261] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 372.684885] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 372.875651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.886424] LNet: Removed LNI 10.1.2.3@tcp [ 372.893465] LNet: Removed LNI 192.168.203.11@tcp [ 372.905117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.942040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 372.962207] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 372.966011] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 373.172086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 373.182884] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 373.620462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.630725] LNet: 44960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.633746] LNet: 44960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 373.636833] LNetError: 44960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 373.640342] LNetError: 44960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 373.644579] LNet: Removed LNI 10.1.2.3@tcp [ 373.646367] LNet: Removed LNI 192.168.203.11@tcp [ 373.655604] LNet: Removed LNI 192.168.203.11@tcp2 [ 373.666554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.696575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 373.707962] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 373.710589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 373.713968] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 376.437988] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:44:39 (1786934679) [ 376.483035] LNet: 45549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.485753] LNetError: 45549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 376.491124] LNetError: 45549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 376.496077] LNet: Removed LNI 10.1.2.3@tcp [ 376.498570] LNet: Removed LNI 192.168.203.11@tcp [ 376.505403] LNet: Removed LNI 192.168.203.11@tcp2 [ 376.801655] Key type .llcrypt unregistered [ 376.804127] Key type ._llcrypt unregistered [ 379.090978] Key type ._llcrypt registered [ 379.092150] Key type .llcrypt registered [ 379.170076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 379.176398] alg: No test for adler32 (adler32-zlib) [ 379.989575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.017351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 380.031477] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 380.409330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.418931] LNet: 45956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.421589] LNetError: 45956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.431550] LNet: Removed LNI 192.168.203.11@tcp [ 380.445750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.479681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 380.493378] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 380.681170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.695714] LNet: Removed LNI 192.168.203.11@tcp [ 380.705755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.733028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 380.750313] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 380.955373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.965482] LNet: 46016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.968774] LNet: 46016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 380.972716] LNetError: 46016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.976961] LNetError: 46016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 380.985197] LNet: Removed LNI 192.168.203.11@tcp [ 380.996779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.027386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 381.041752] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 381.240865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 381.299258] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 381.302443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 381.304467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 383.484296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.493556] LNet: 46095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.497571] LNetError: 46095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 383.505691] LNet: Removed LNI 192.168.203.11@tcp [ 383.517155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.554274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 383.574691] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 386.244825] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:44:49 (1786934689) [ 386.297203] LNet: 46687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.302570] LNetError: 46687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 386.312288] LNet: Removed LNI 192.168.203.11@tcp [ 386.598845] Key type .llcrypt unregistered [ 386.601382] Key type ._llcrypt unregistered [ 389.004377] Key type ._llcrypt registered [ 389.006144] Key type .llcrypt registered [ 389.090859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 389.097828] alg: No test for adler32 (adler32-zlib) [ 389.913096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.945843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 389.959743] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 390.336632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.350443] LNet: 47093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.353846] LNetError: 47093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 390.364599] LNet: Removed LNI 192.168.203.11@tcp [ 390.374379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.406761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 390.423851] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 393.192226] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:44:55 (1786934695) [ 393.231352] LNet: 47677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.234568] LNetError: 47677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.242860] LNet: Removed LNI 192.168.203.11@tcp [ 393.521857] Key type .llcrypt unregistered [ 393.523778] Key type ._llcrypt unregistered [ 395.888856] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 395.958426] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 395.963215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 395.966872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 398.179568] Key type ._llcrypt registered [ 398.181279] Key type .llcrypt registered [ 398.251971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 398.260030] alg: No test for adler32 (adler32-zlib) [ 399.079442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.111991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 399.124459] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 399.510744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.520749] LNet: 48145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.523484] LNetError: 48145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 399.532441] LNet: Removed LNI 192.168.203.11@tcp [ 399.543327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.573875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 399.587875] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 402.019597] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:45:04 (1786934704) [ 402.069812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.079748] LNet: 48739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.083285] LNetError: 48739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 402.093873] LNet: Removed LNI 192.168.203.11@tcp [ 402.104477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.144534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 404.576698] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:45:07 (1786934707) [ 404.611319] LNet: 49316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.857615] Key type .llcrypt unregistered [ 404.859164] Key type ._llcrypt unregistered [ 407.078257] Key type ._llcrypt registered [ 407.079888] Key type .llcrypt registered [ 407.151431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 407.158539] alg: No test for adler32 (adler32-zlib) [ 407.977552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.003726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 408.016987] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 408.382942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.395195] LNet: 49724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.398709] LNetError: 49724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.412935] LNet: Removed LNI 192.168.203.11@tcp [ 408.424942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.453703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 408.465926] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 411.176590] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:45:13 (1786934713) [ 411.235542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.252727] LNet: 50314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.255926] LNetError: 50314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.269797] LNet: Removed LNI 192.168.203.11@tcp [ 411.281349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.314675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 413.688131] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:45:16 (1786934716) [ 414.355946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 415.027543] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:45:17 (1786934717) [ 415.640816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 416.335610] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:45:19 (1786934719) [ 416.964742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 417.630339] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:45:20 (1786934720) [ 418.226776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 418.969194] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:45:21 (1786934721) [ 419.601326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 420.257866] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:45:23 (1786934723) [ 420.293252] LNet: 52359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.573682] Key type .llcrypt unregistered [ 420.575150] Key type ._llcrypt unregistered [ 422.709596] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 422.762417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 422.765637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 422.769727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 424.999743] Key type ._llcrypt registered [ 425.001237] Key type .llcrypt registered [ 425.070665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 425.076074] alg: No test for adler32 (adler32-zlib) [ 425.881522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.912616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 425.922825] LNet: Added LNI 192.168.203.11@tcp [8/512/0/180] [ 425.926505] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 425.928696] LNet: 52689:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 425.972903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.982877] LNet: 52708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.986631] LNetError: 52708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 425.990078] LNet: Removed LNI 10.1.2.3@tcp [ 425.991529] LNetError: 52708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.001548] LNet: Removed LNI 192.168.203.11@tcp [ 426.011116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.043255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 426.054418] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 426.083684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.093229] LNet: 52732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.096601] LNetError: 52732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.105326] LNet: Removed LNI 192.168.203.11@tcp [ 426.114304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.146329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 426.158796] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 426.161251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 426.162880] LNet: 52740:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 426.204931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.215452] LNet: 52758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.219501] LNetError: 52758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 426.223716] LNet: Removed LNI 10.1.2.3@tcp [ 426.225355] LNetError: 52758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.237837] LNet: Removed LNI 192.168.203.11@tcp [ 426.247925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.278798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 426.291796] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 426.295924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 426.297846] LNet: 52766:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 426.370440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.382061] LNet: 52792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.386089] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 426.389971] LNet: Removed LNI 10.1.2.3@tcp [ 426.391412] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.401454] LNet: Removed LNI 192.168.203.11@tcp [ 426.410570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.438092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 426.449339] LNet: Added LNI 192.168.203.11@tcp [8/256/0/182] [ 426.452540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 426.487867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.500309] LNet: 52820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.504197] LNetError: 52820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 426.507416] LNet: Removed LNI 10.1.2.3@tcp [ 426.509188] LNetError: 52820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.516335] LNet: Removed LNI 192.168.203.11@tcp [ 426.526252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.556544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 426.567570] LNet: Added LNI 192.168.203.11@tcp [8/256/0/182] [ 426.569585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 426.571282] LNet: 52828:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 426.600679] LNet: Removed LNI 10.1.2.3@tcp [ 426.608439] LNet: Removed LNI 192.168.203.11@tcp [ 426.863169] Key type .llcrypt unregistered [ 426.865115] Key type ._llcrypt unregistered [ 431.026720] Key type ._llcrypt registered [ 431.029368] Key type .llcrypt registered [ 431.590978] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:45:34 (1786934734) [ 432.164492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 432.835566] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:45:35 (1786934735) [ 432.984194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 432.991447] alg: No test for adler32 (adler32-zlib) [ 433.807550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.832824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 433.847383] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 434.150039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.11 [ 434.160805] LNet: 54094:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 434.166044] LNet: Removed LNI 192.168.203.11@tcp [ 436.535718] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:45:39 (1786934739) [ 436.612662] LNet: 54663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.615732] LNetError: 54663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 436.624394] LNet: Removed LNI 192.168.203.11@tcp [ 436.897425] Key type .llcrypt unregistered [ 436.899069] Key type ._llcrypt unregistered [ 439.141474] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 439.251739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 441.572892] Key type ._llcrypt registered [ 441.574273] Key type .llcrypt registered [ 441.647142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 441.654127] alg: No test for adler32 (adler32-zlib) [ 442.645700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 445.453959] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:45:48 (1786934748) [ 445.508210] LNet: 55575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.513695] LNetError: 55575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 445.523804] LNet: Removed LNI 10.1.2.3@tcp [ 445.801492] Key type .llcrypt unregistered [ 445.803703] Key type ._llcrypt unregistered [ 448.184869] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 448.280492] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 448.283350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 448.286122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 450.589271] Key type ._llcrypt registered [ 450.590781] Key type .llcrypt registered [ 450.658846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 450.667078] alg: No test for adler32 (adler32-zlib) [ 451.495290] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 453.855605] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:45:56 (1786934756) [ 454.413118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 455.037158] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:45:57 (1786934757) [ 455.071821] LNet: 56757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.076559] LNetError: 56757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 455.084621] LNet: Removed LNI 10.1.2.3@tcp [ 455.343734] Key type .llcrypt unregistered [ 455.344827] Key type ._llcrypt unregistered [ 457.536397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 457.643470] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 457.646079] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 457.649789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 459.930522] Key type ._llcrypt registered [ 459.931683] Key type .llcrypt registered [ 460.001298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 460.007915] alg: No test for adler32 (adler32-zlib) [ 460.811419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.842280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 463.833903] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:46:06 (1786934766) [ 464.012076] LNet: 57998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.015521] LNetError: 57998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 464.029366] LNet: Removed LNI 10.1.2.3@tcp [ 464.304793] Key type .llcrypt unregistered [ 464.306296] Key type ._llcrypt unregistered [ 466.440087] Key type ._llcrypt registered [ 466.441815] Key type .llcrypt registered [ 466.536161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 466.545104] alg: No test for adler32 (adler32-zlib) [ 467.358626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 467.374105] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 470.419946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 470.668405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_interrupt -r 1 [ 470.687991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_interrupt -r 1 [ 470.704307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 470.712691] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 470.784313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 470.797645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 470.813987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 470.827672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 470.840959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 470.855222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 470.869155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 470.882751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 470.890752] LNet: 58438:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 470.900505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 470.914621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 470.927423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 470.941655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 470.956668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 470.972245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 470.987635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 471.180815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_dropped -r 1 [ 471.195248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_dropped -r 1 [ 471.210267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 471.219448] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 471.299779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 471.311490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 471.323942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 471.338174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 471.351910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 471.367231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 471.383553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 471.397630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 471.404875] LNet: 58547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 471.412408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 471.423418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 471.436797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 471.448677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 471.456314] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 471.463406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 471.480030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 471.496070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 471.706459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_aborted -r 1 [ 471.726760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_aborted -r 1 [ 471.743495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 471.846135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 471.862359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 471.880981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 471.896671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 471.913783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 471.927839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 471.941820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 471.959219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 471.975200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 471.989854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 472.003339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 472.019206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 472.032850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 472.046672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 472.060686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 472.282778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_no_route -r 1 [ 472.300958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_no_route -r 1 [ 472.319768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 472.329186] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 472.332614] LNet: Skipped 2 previous similar messages [ 472.409833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 472.422404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 472.436867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 472.451609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 472.465676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 472.478531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 472.493584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 472.508444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 472.518455] LNet: 58765:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 472.522890] LNet: 58765:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 472.532987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 472.549496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 472.565478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 472.581101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 472.595156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 472.609130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 472.623583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 472.826462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_timeout -r 1 [ 472.845136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_timeout -r 1 [ 472.863048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 472.968667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 472.983313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 473.000838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 473.017645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 473.033225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 473.047545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 473.061635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 473.075403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 473.088533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 473.103447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 473.118350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 473.133245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 473.147854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 473.162828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 473.177933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 473.399389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_error -r 1 [ 473.416321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_error -r 1 [ 473.432201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 473.517043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 473.530903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 473.543385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 473.556920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 473.572340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 473.586591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 473.599639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 473.612881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 473.626241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 473.640455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 473.656253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 473.671907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 473.687972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 473.703794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 473.720163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 473.832077] LNet: 59028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.835635] LNetError: 59028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 473.844570] LNet: Removed LNI 192.168.203.11@tcp [ 474.002416] Key type .llcrypt unregistered [ 474.003992] Key type ._llcrypt unregistered [ 479.402357] Key type ._llcrypt registered [ 479.403735] Key type .llcrypt registered [ 479.857042] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:46:22 (1786934782) [ 480.157421] Key type .llcrypt unregistered [ 480.158738] Key type ._llcrypt unregistered [ 482.160677] Key type ._llcrypt registered [ 482.162257] Key type .llcrypt registered [ 482.227156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 482.232273] alg: No test for adler32 (adler32-zlib) [ 483.034351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 483.046055] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 485.871367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 489.075481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.087667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 489.096435] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 489.544748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_interrupt -r 1 [ 489.560729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 1 [ 489.577939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_interrupt -r 1 [ 489.592561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 1 [ 489.607881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_interrupt -r 1 [ 489.621798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 1 [ 489.639540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_interrupt -r 1 [ 489.653756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 1 [ 489.669422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 489.679285] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 489.682921] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 489.757495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 489.769268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 489.781894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 489.793968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 489.808258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 489.821921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 489.835126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 489.847769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 489.856740] LNet: 60493:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 489.864564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 489.878625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 489.893801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 489.906483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 489.919696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 489.934821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 489.951716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 490.126715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_dropped -r 1 [ 490.143478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 1 [ 490.158514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_dropped -r 1 [ 490.171825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 1 [ 490.185115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_dropped -r 1 [ 490.199627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 1 [ 490.215074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_dropped -r 1 [ 490.231964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 1 [ 490.246229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 490.254596] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 490.256871] LNet: Skipped 2 previous similar messages [ 490.324821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 490.335601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 490.348229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 490.360569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 490.374237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 490.387937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 490.402577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 490.415567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 490.421804] LNet: 60615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 490.429288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 490.440874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 490.451590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 490.461916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 490.474454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 490.487291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 490.503065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 490.677901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_aborted -r 1 [ 490.690601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 1 [ 490.702714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_aborted -r 1 [ 490.714548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 1 [ 490.725416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_aborted -r 1 [ 490.736899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 1 [ 490.751512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_aborted -r 1 [ 490.769250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 1 [ 490.786452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 490.858669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 490.870205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 490.880333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 490.890091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 490.902729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 490.915437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 490.927247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 490.938055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 490.951216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 490.964979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 490.977443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 490.988624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 490.998829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 491.012682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 491.024548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 491.181706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_no_route -r 1 [ 491.195183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 1 [ 491.209737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_no_route -r 1 [ 491.224339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 1 [ 491.239635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_no_route -r 1 [ 491.254509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 1 [ 491.267188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_no_route -r 1 [ 491.279568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 1 [ 491.293573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 491.300666] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 491.303396] LNet: Skipped 6 previous similar messages [ 491.369204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 491.381313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 491.393338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 491.405579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 491.419403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 491.430511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 491.443864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 491.455221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 491.461935] LNet: 60857:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 491.464423] LNet: 60857:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 491.473667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 491.487462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 491.500959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 491.513834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 491.526117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 491.537547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 491.550020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 491.712260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_timeout -r 1 [ 491.727164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 1 [ 491.743178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_timeout -r 1 [ 491.758984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 1 [ 491.774945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_timeout -r 1 [ 491.790044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 1 [ 491.803846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_timeout -r 1 [ 491.820581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 1 [ 491.834339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 491.909615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 491.920515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 491.930776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 491.941804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 491.955445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 491.967868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 491.981039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 491.994447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 492.008362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 492.020180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 492.031386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 492.042083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 492.053676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 492.066197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 492.080196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 492.249698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_error -r 1 [ 492.262704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_error -r 1 [ 492.276448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_error -r 1 [ 492.290361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_error -r 1 [ 492.306344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_error -r 1 [ 492.321295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_error -r 1 [ 492.335767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_error -r 1 [ 492.351841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_error -r 1 [ 492.368225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 492.448984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 492.461946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 492.474475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 492.488288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 492.501691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 492.512893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 492.524356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 492.535876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 492.548936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 492.563293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 492.576682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 492.589507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 492.602435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 492.616977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 492.630749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 494.778423] LNet: 61179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.781583] LNetError: 61179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 494.785015] LNet: Removed LNI 192.168.203.11@tcp [ 495.846429] LNet: Removed LNI 192.168.203.11@tcp1 [ 495.983473] Key type .llcrypt unregistered [ 495.985329] Key type ._llcrypt unregistered [ 501.807477] Key type ._llcrypt registered [ 501.809115] Key type .llcrypt registered [ 502.301636] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:46:45 (1786934805) [ 502.638293] Key type .llcrypt unregistered [ 502.639992] Key type ._llcrypt unregistered [ 504.905086] Key type ._llcrypt registered [ 504.906706] Key type .llcrypt registered [ 504.977211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 504.984336] alg: No test for adler32 (adler32-zlib) [ 505.793612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 505.805783] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 508.803508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 509.054400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_dropped -r 1 [ 509.070916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_dropped -r 1 [ 509.084779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 509.093061] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 509.175282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 509.190139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 509.205170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 509.219237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 509.231982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 509.247704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 509.263903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 509.280089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 509.289908] LNet: 62428:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 509.300705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 509.316933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 509.331861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 509.348066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 509.364652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 509.382620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 509.404484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 509.583867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_error -r 1 [ 509.598457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_error -r 1 [ 509.615051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 509.623840] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 509.703524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 509.719615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 509.734064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 509.749356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 509.763432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 509.778916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 509.794465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 509.808402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 509.816902] LNet: 62537:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 509.826967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 509.841861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 509.856522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 509.870152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 509.884441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 509.897038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 509.912370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 510.110875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_timeout -r 1 [ 510.129124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_timeout -r 1 [ 510.146249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 510.230027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 510.243806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 510.256884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 510.270935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 510.285623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 510.299494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 510.312868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 510.326393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 510.342637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 510.357227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 510.370821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 510.386617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 510.401863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 510.417588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 510.432959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 510.542551] LNet: 62691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.546700] LNetError: 62691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.558284] LNet: Removed LNI 192.168.203.11@tcp [ 510.707589] Key type .llcrypt unregistered [ 510.709447] Key type ._llcrypt unregistered [ 516.139294] Key type ._llcrypt registered [ 516.140430] Key type .llcrypt registered [ 516.592752] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:46:59 (1786934819) [ 516.895409] Key type .llcrypt unregistered [ 516.896662] Key type ._llcrypt unregistered [ 518.934383] Key type ._llcrypt registered [ 518.935857] Key type .llcrypt registered [ 519.004594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 519.011375] alg: No test for adler32 (adler32-zlib) [ 519.817865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 519.829281] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 522.741495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 525.611787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.627296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 525.638136] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 526.171307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_dropped -r 1 [ 526.189303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 1 [ 526.208242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_dropped -r 1 [ 526.226801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 1 [ 526.244861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_dropped -r 1 [ 526.262716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 1 [ 526.283903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_dropped -r 1 [ 526.305987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 1 [ 526.325138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 526.335835] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 526.339296] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.111@tcp [ 526.424345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 526.439946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 526.454121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 526.468162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 526.482733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 526.497251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 526.511673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 526.528882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 526.540808] LNet: 64156:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 526.549663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 526.564223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 526.578989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 526.592609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 526.609192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 526.622980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 526.638859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 526.652948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 526.853712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_error -r 1 [ 526.871844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_error -r 1 [ 526.887840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_error -r 1 [ 526.901799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_error -r 1 [ 526.917963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_error -r 1 [ 526.932524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_error -r 1 [ 526.948474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_error -r 1 [ 526.964903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_error -r 1 [ 526.983795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 526.992865] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 526.995820] LNet: Skipped 2 previous similar messages [ 527.080196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 527.094286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 527.107469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 527.120643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 527.131796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 527.147342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 527.162431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 527.178048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 527.187372] LNet: 64279:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 527.197348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 527.213073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 527.228681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 527.244769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 527.260362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 527.277493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 527.293153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 527.308805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 527.517183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_timeout -r 1 [ 527.534117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_timeout -r 1 [ 527.550223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_timeout -r 1 [ 527.565665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_timeout -r 1 [ 527.579770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_timeout -r 1 [ 527.598913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_timeout -r 1 [ 527.615197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_timeout -r 1 [ 527.630146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_timeout -r 1 [ 527.648163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 527.748057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 527.764588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 527.780150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 527.800720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 527.813911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 527.830997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 527.849444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 527.862405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 527.876031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 527.889229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 527.902308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 527.915178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 527.929704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 527.943741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 527.959638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 527.973497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 530.171263] LNet: 64484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 530.175156] LNetError: 64484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 530.178883] LNet: Removed LNI 192.168.203.11@tcp [ 530.186768] LNet: Removed LNI 192.168.203.11@tcp1 [ 530.349418] Key type .llcrypt unregistered [ 530.350502] Key type ._llcrypt unregistered [ 535.537628] Key type ._llcrypt registered [ 535.538667] Key type .llcrypt registered [ 535.940390] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:47:18 (1786934838) [ 536.413424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 536.940089] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:47:19 (1786934839) [ 537.193260] Key type .llcrypt unregistered [ 537.194869] Key type ._llcrypt unregistered [ 539.229904] Key type ._llcrypt registered [ 539.231477] Key type .llcrypt registered [ 539.304940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 539.311478] alg: No test for adler32 (adler32-zlib) [ 540.123269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 540.135209] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 543.110774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 543.317212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e network_timeout -r 1 [ 543.331148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e network_timeout -r 1 [ 543.348964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 543.359200] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 543.436280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 543.451244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 543.466241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 543.481560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 543.497094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 543.511795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 543.524977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 543.539068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 543.549252] LNet: 66027:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 543.559182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 543.572255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 543.583949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 543.596425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 543.609791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 543.624441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 543.730986] LNet: 66070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.734673] LNetError: 66070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.742360] LNet: Removed LNI 192.168.203.11@tcp [ 543.881402] Key type .llcrypt unregistered [ 543.883247] Key type ._llcrypt unregistered [ 550.091647] Key type ._llcrypt registered [ 550.093259] Key type .llcrypt registered [ 550.162721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 550.169383] alg: No test for adler32 (adler32-zlib) [ 550.974098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 550.988117] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 553.933739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 556.939569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.954676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 556.965172] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 557.479628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e network_timeout -r 1 [ 557.494495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e network_timeout -r 1 [ 557.512505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e network_timeout -r 1 [ 557.530453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e network_timeout -r 1 [ 557.548091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e network_timeout -r 1 [ 557.566958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e network_timeout -r 1 [ 557.585524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e network_timeout -r 1 [ 557.603261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e network_timeout -r 1 [ 557.621514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 557.631276] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 557.715101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 557.727543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 557.740529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 557.753181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 557.764584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 557.777973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 557.789791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 557.805894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 557.816243] LNet: 66949:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 557.826392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 557.841135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 557.855884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 557.870966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 557.885758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 557.901257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 560.011622] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 560.115780] LNet: 67026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.118722] LNetError: 67026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.121846] LNet: Removed LNI 192.168.203.11@tcp [ 560.128661] LNet: Removed LNI 192.168.203.11@tcp1 [ 560.266430] Key type .llcrypt unregistered [ 560.267677] Key type ._llcrypt unregistered [ 565.606530] Key type ._llcrypt registered [ 565.608330] Key type .llcrypt registered [ 566.057681] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:47:48 (1786934868) [ 566.187513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 566.194474] alg: No test for adler32 (adler32-zlib) [ 566.999301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.024532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 567.036431] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 567.290801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 567.299158] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 567.622787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 567.649263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 567.665824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 567.681034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 567.696967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 567.713809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 567.729190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.11@tcp [ 567.738776] LNetError: 68123:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 568.032395] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 568.032482] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 570.080303] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 570.892417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 574.176323] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 575.106112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 575.123065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 575.148681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.158208] LNet: 68240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.161781] LNetError: 68240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.165589] LNet: Removed LNI 192.168.203.11@tcp [ 575.176335] LNet: Removed LNI 192.168.203.11@tcp1 [ 575.185548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.210697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 575.221876] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 575.499689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 575.507764] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 575.849888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 575.867404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 575.894956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 575.917873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 575.936429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 575.957231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 575.971467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 575.985924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.11@tcp [ 575.995121] LNetError: 68512:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 576.224319] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 579.164758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 582.368347] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 582.372016] LNet: Skipped 1 previous similar message [ 587.531427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 587.548773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 587.566260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.129670] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:48:12 (1786934892) [ 590.166637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.178212] LNet: 69233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.182091] LNetError: 69233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.185047] LNetError: 69233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 590.188544] LNet: Removed LNI 192.168.203.11@tcp [ 590.198671] LNet: Removed LNI 192.168.203.11@tcp1 [ 590.209517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.231814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.244265] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 590.535744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 590.546255] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 590.902056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 590.937850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 590.953214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 590.972526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 590.993387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 591.010851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 591.029089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.11@tcp --health 0 [ 591.073250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 591.264194] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.11@tcp [ 592.288618] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 592.291433] LNet: Skipped 1 previous similar message [ 607.551597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 607.599746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 607.614319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 607.628330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.11@tcp --health 500 [ 607.663180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 613.848287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 613.873127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.882543] LNet: 69746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.885315] LNetError: 69746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.888702] LNetError: 69746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 613.892459] LNet: Removed LNI 192.168.203.11@tcp [ 613.903317] LNet: Removed LNI 192.168.203.11@tcp1 [ 613.912633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.937044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.948818] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 614.223127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 614.232379] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 614.549382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 614.566331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 614.590148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 614.604196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 614.618941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 614.635053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 614.650461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 614.666627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.11@tcp --health 0 [ 615.968312] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 615.971111] LNet: Skipped 3 previous similar messages [ 616.821137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 627.238014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 627.251832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 627.265473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 629.630652] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:48:52 (1786934932) [ 629.796783] LNet: 70812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.800273] LNetError: 70812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.803623] LNetError: 70812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 629.806413] LNet: Removed LNI 192.168.203.11@tcp [ 629.820491] LNet: Removed LNI 192.168.203.11@tcp1 [ 630.069837] Key type .llcrypt unregistered [ 630.071359] Key type ._llcrypt unregistered [ 632.161061] Key type ._llcrypt registered [ 632.162656] Key type .llcrypt registered [ 632.229618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 632.235143] alg: No test for adler32 (adler32-zlib) [ 633.036588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 633.047835] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 636.416118] Lustre: DEBUG MARKER: Initial discovery [ 636.435330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.111@tcp [ 637.223443] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 637.268084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 637.286204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.111@tcp [ 637.295301] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 637.304245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 637.318671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.111@tcp -r 1 -m GET -l 3 [ 637.334809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.111@tcp -r 1 -m GET -e local_error [ 637.350626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.111@tcp -r 1 -m PUT -l 6 [ 637.832672] Lustre: DEBUG MARKER: Force oleg311-client.virtnet to discover 192.168.203.111@tcp (in background) [ 637.874584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.111@tcp [ 638.176325] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 638.345460] Lustre: DEBUG MARKER: Force oleg311-server to discover 192.168.203.11@tcp [ 639.120314] Lustre: DEBUG MARKER: Wait for 71574 [ 640.928243] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 641.427116] Lustre: DEBUG MARKER: Finished wait on 71574 [ 641.456518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 641.472617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 641.581214] LNet: 72030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.583682] LNetError: 72030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.592453] LNet: Removed LNI 192.168.203.11@tcp [ 641.732421] Key type .llcrypt unregistered [ 641.733778] Key type ._llcrypt unregistered [ 647.196249] Key type ._llcrypt registered [ 647.197619] Key type .llcrypt registered [ 647.665490] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:49:10 (1786934950) [ 647.839483] Key type .llcrypt unregistered [ 647.840806] Key type ._llcrypt unregistered [ 649.933515] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 649.983099] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 649.986896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 649.989060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 652.210482] Key type ._llcrypt registered [ 652.212310] Key type .llcrypt registered [ 652.283069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 652.289932] alg: No test for adler32 (adler32-zlib) [ 653.098912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.122728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.133031] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 653.412750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 653.422304] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 656.406526] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:49:19 (1786934959) [ 656.452600] LNet: 74105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.457760] LNetError: 74105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 656.461142] LNet: Removed LNI 10.1.2.3@tcp [ 656.468830] LNet: Removed LNI 192.168.203.11@tcp [ 656.723362] Key type .llcrypt unregistered [ 656.725018] Key type ._llcrypt unregistered [ 659.014822] Key type ._llcrypt registered [ 659.016593] Key type .llcrypt registered [ 659.098550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 659.106468] alg: No test for adler32 (adler32-zlib) [ 659.913864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.940825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.953987] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 660.226161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 660.236610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 660.598134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.11@tcp [ 661.664290] LNetError: Unexpected error -22 connecting to 192.168.203.11@tcp at host 192.168.203.11:988 [ 662.807502] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:49:25 (1786934965) [ 662.853959] LNet: 75242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.858393] LNetError: 75242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 662.861960] LNet: Removed LNI 10.1.2.3@tcp [ 662.869769] LNet: Removed LNI 192.168.203.11@tcp [ 663.145683] Key type .llcrypt unregistered [ 663.147081] Key type ._llcrypt unregistered [ 665.361550] Key type ._llcrypt registered [ 665.363327] Key type .llcrypt registered [ 665.468308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 665.477747] alg: No test for adler32 (adler32-zlib) [ 666.283360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.307956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 666.321412] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 666.603448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 666.613245] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 666.967366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.11@tcp1 --nid 192.168.203.11@tcp2 [ 668.511606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp1 192.168.203.11@tcp2 [ 668.520895] LNetError: 76083:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.11@tcp1 to 192.168.203.11@tcp2 - Route table may be misconfigured [ 668.531610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp2 192.168.203.11@tcp1 [ 670.782832] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:49:33 (1786934973) [ 670.817588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.828628] LNet: 76651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.832888] LNetError: 76651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.836681] LNet: Removed LNI 192.168.203.11@tcp1 [ 671.846519] LNet: Removed LNI 192.168.203.11@tcp2 [ 671.855646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.879503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.890161] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 672.171933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 672.182345] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 672.509111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 672.524821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.11@tcp [ 672.538886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp1 192.168.203.11@tcp1 [ 672.554859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp -d 192.168.203.11@tcp -e network_timeout [ 672.570623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -e network_timeout [ 672.587754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -e network_timeout [ 672.605855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -e network_timeout [ 672.625040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 672.637230] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 672.647388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 672.686896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 672.731235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 672.929083] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 674.874608] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:49:37 (1786934977) [ 674.910421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.920321] LNet: 77513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.924071] LNetError: 77513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.927746] LNetError: 77513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 674.931255] LNet: Removed LNI 192.168.203.11@tcp [ 675.942418] LNet: Removed LNI 192.168.203.11@tcp1 [ 675.950733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.979410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 676.078909] LNet: 77559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.219459] Key type .llcrypt unregistered [ 676.221030] Key type ._llcrypt unregistered [ 680.279384] Key type ._llcrypt registered [ 680.281078] Key type .llcrypt registered [ 680.768394] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:49:43 (1786934983) [ 680.903490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.909737] alg: No test for adler32 (adler32-zlib) [ 681.714589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.237244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 682.810247] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:49:45 (1786934985) [ 682.847763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.858232] LNet: 78651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.869537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.894439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.906113] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 683.188154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 683.197592] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 683.510273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 683.526280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp1 [ 683.542290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp1 [ 685.607761] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:49:48 (1786934988) [ 686.198372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg311-server [ 686.746547] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:49:49 (1786934989) [ 687.359463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg311-server [ 687.954198] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:49:50 (1786934990) [ 688.590490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg311-server [ 689.205275] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:49:52 (1786934992) [ 689.835917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg311-server [ 690.391614] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:49:53 (1786934993) [ 690.983834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg311-server [ 691.561416] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:49:54 (1786934994) [ 692.180148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg311-server [ 692.755883] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:49:55 (1786934995) [ 693.414433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg311-server [ 694.018680] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:49:56 (1786934996) [ 694.663586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg311-server [ 695.250696] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:49:58 (1786934998) [ 695.893154] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg311-server [ 696.487588] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:49:59 (1786934999) [ 696.515839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.526196] LNet: 82522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.529530] LNetError: 82522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.533270] LNet: Removed LNI 192.168.203.11@tcp [ 697.574402] LNet: Removed LNI 192.168.203.11@tcp1 [ 697.583101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.607283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.618790] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 697.887499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 697.919557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 697.990195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.001179] LNet: 82674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.004498] LNetError: 82674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.007932] LNetError: 82674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 698.019510] LNet: Removed LNI 192.168.203.11@tcp [ 698.027669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.050842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 698.062467] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 698.351467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 698.419447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.494338] LNet: Removed LNI 192.168.203.11@tcp [ 699.501792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.520816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.530610] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 699.778464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 699.807445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 699.877453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.887373] LNetError: 82967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.890680] LNetError: 82967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 699.900494] LNet: Removed LNI 192.168.203.11@tcp [ 699.910097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.931562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 699.943439] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 700.227579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 700.296873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.305957] LNet: 83112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.309125] LNet: 83112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 701.350365] LNet: Removed LNI 192.168.203.11@tcp [ 701.358543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.382607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.394793] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 701.660882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 701.692877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 701.767617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.822287] LNet: Removed LNI 192.168.203.11@tcp [ 702.830609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.851619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 702.862909] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 703.134833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 703.207124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.215954] LNetError: 83404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.218502] LNetError: 83404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 703.225376] LNet: Removed LNI 192.168.203.11@tcp [ 703.234604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.260621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 703.274251] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 703.537571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 703.571193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 703.647731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.678587] LNet: Removed LNI 192.168.203.11@tcp [ 704.689040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.715815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 704.728322] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 705.021668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 705.099415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.108487] LNet: 83696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.111728] LNet: 83696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 706.150373] LNet: Removed LNI 192.168.203.11@tcp [ 706.159225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.181916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.193410] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 706.487027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 706.520857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 706.612505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.686486] LNet: Removed LNI 192.168.203.11@tcp [ 707.695718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.723470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 707.734440] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 708.019183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 708.106923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.116924] LNetError: 83989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.119647] LNetError: 83989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 708.127421] LNet: Removed LNI 192.168.203.11@tcp [ 708.135299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.154078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 708.163372] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 708.441781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 708.471922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 709.612502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.630589] LNet: Removed LNI 192.168.203.11@tcp [ 710.639321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.658455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 710.667960] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 710.934684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 711.018430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.038436] LNet: Removed LNI 192.168.203.11@tcp [ 712.046082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.067373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.078957] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 712.329951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 712.352811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.368327] LNet: Removed LNI 192.168.203.11@tcp [ 712.376393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.400318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.411661] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 712.668880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 712.703488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.722369] LNet: Removed LNI 192.168.203.11@tcp [ 712.731163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.752524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 712.784314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.799654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.821884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 712.849855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.862946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.883979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 712.916780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.931934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.954433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 712.965067] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 713.224542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.233928] LNet: 84729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.236640] LNet: 84729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 713.243285] LNet: Removed LNI 192.168.203.11@tcp [ 713.251803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.279967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.617397] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:50:18 (1786935018) [ 715.649139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.670102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.696606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 715.707073] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 716.127310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.135924] LNetError: 85448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.138563] LNetError: 85448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 716.151279] LNet: Removed LNI 192.168.203.11@tcp231 [ 716.158126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.182750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 716.193475] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 716.195769] LNet: 85456:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 716.337165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 716.355733] LNet: Removed LNI 192.168.203.11@tcp231 [ 716.362238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 716.371339] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 716.743133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.756469] LNet: Removed LNI 192.168.203.11@tcp231 [ 716.763451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.785891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 716.795673] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/0] [ 716.798482] LNet: 85608:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 716.942800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 716.959178] LNet: Removed LNI 192.168.203.11@tcp231 [ 716.968227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 716.979898] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/0] [ 717.358513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.374299] LNet: Removed LNI 192.168.203.11@tcp231 [ 717.382018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.408501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 717.421545] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/60] [ 717.562113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 717.584305] LNet: Removed LNI 192.168.203.11@tcp231 [ 717.591027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 717.601243] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/60] [ 718.003523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.016359] LNet: Removed LNI 192.168.203.11@tcp231 [ 718.023717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.050639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 718.060870] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 718.063661] LNet: 85910:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 718.067764] LNet: 85910:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 718.215065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 718.231928] LNet: Removed LNI 192.168.203.11@tcp231 [ 718.239436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 718.251505] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 720.856723] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:50:23 (1786935023) [ 720.895316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.909763] LNet: Removed LNI 192.168.203.11@tcp231 [ 720.920283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.948028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 720.959869] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 721.232135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 721.245505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 721.260537] LNet: Removed LNI 192.168.203.11@tcp232 [ 721.267131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 721.278425] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 721.531777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 721.547230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 721.565592] LNet: Removed LNI 192.168.203.11@tcp232 [ 721.572738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 721.584111] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 721.864109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 721.882165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 721.899543] LNet: Removed LNI 192.168.203.11@tcp232 [ 721.907229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 721.917862] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 722.177878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 722.194760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 722.213486] LNet: Removed LNI 192.168.203.11@tcp232 [ 722.220166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 724.275235] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:50:27 (1786935027) [ 724.648390] Key type .llcrypt unregistered [ 724.650124] Key type ._llcrypt unregistered [ 726.685374] Key type ._llcrypt registered [ 726.686718] Key type .llcrypt registered [ 726.749065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.754764] alg: No test for adler32 (adler32-zlib) [ 727.557512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 727.568350] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 730.437551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 733.197042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.211375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 733.220822] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 733.756858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_interrupt -r 2 [ 733.768880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 2 [ 733.780125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_interrupt -r 2 [ 733.793486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 2 [ 733.809397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_interrupt -r 2 [ 733.825416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 2 [ 733.842113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_interrupt -r 2 [ 733.858675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 2 [ 733.877903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 733.888691] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 733.891371] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 733.967875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 733.980017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 733.992980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 734.005760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 734.018965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 734.031902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.044491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.057533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.066541] LNet: 88410:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 734.074902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 734.088689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 734.102324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 734.115431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 734.128191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 734.140050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 734.318603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_dropped -r 2 [ 734.336522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 2 [ 734.354756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_dropped -r 2 [ 734.372833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 2 [ 734.390313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_dropped -r 2 [ 734.406126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 2 [ 734.421518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_dropped -r 2 [ 734.438973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 2 [ 734.456490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 734.465402] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 734.558178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 734.571494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 734.583887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 734.598345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 734.612663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 734.626835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 734.640426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 734.654231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 734.663254] LNet: 88530:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 734.671713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 734.684313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 734.696478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 734.709843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 734.724797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 734.739770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 734.914653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_aborted -r 2 [ 734.929683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 2 [ 734.947318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_aborted -r 2 [ 734.964023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 2 [ 734.981160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_aborted -r 2 [ 735.000495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 2 [ 735.017325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_aborted -r 2 [ 735.034432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 2 [ 735.052762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 735.168311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 735.179657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 735.192608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 735.205366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 735.219995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 735.233051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 735.247266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 735.262696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 735.276057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.288488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.302885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.316645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 735.331135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 735.347802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 735.544924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_no_route -r 2 [ 735.560602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 2 [ 735.577186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_no_route -r 2 [ 735.594457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 2 [ 735.614641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_no_route -r 2 [ 735.633313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 2 [ 735.652023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_no_route -r 2 [ 735.667272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 2 [ 735.684895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 735.696893] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 735.700187] LNet: Skipped 2 previous similar messages [ 735.792619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 735.806193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 735.822502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 735.838203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 735.852097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 735.866097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 735.881077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 735.897056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 735.905099] LNet: 88770:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 735.908800] LNet: 88770:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 735.917794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 735.931319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 735.944521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 735.958924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 735.974251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 735.989839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 736.175609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_timeout -r 2 [ 736.192379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 2 [ 736.207393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_timeout -r 2 [ 736.225121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 2 [ 736.242017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_timeout -r 2 [ 736.258119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 2 [ 736.274138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_timeout -r 2 [ 736.292128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 2 [ 736.310213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 736.404285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 736.419819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 736.433487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 736.445952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 736.459873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 736.474383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 736.487690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 736.500859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 736.515392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 736.530761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.544265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 736.558353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 736.572165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 736.588899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 736.785613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_dropped -r 2 [ 736.800298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 2 [ 736.815984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_dropped -r 2 [ 736.833497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 2 [ 736.850568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_dropped -r 2 [ 736.865779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 2 [ 736.882077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_dropped -r 2 [ 736.899135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 2 [ 736.915847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 736.925086] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.111@tcp [ 737.003229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 737.017095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 737.030414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 737.043374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 737.058147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 737.070875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 737.085640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 737.100160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 737.116031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 737.130539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 737.142433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 737.156572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 737.171602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 737.186494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 739.452792] LNet: 89088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.455826] LNetError: 89088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.459163] LNet: Removed LNI 192.168.203.11@tcp [ 739.467527] LNet: Removed LNI 192.168.203.11@tcp1 [ 739.625547] Key type .llcrypt unregistered [ 739.627266] Key type ._llcrypt unregistered [ 745.118815] Key type ._llcrypt registered [ 745.120150] Key type .llcrypt registered [ 745.574477] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:50:48 (1786935048) [ 745.692761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.698956] alg: No test for adler32 (adler32-zlib) [ 746.505620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.525642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.536508] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 746.841234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 746.853332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.11@tcp [ 746.862320] LNetError: 90064:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 746.894673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.11@tcp [ 746.933084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 749.041826] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:50:51 (1786935051) [ 749.071227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.080108] LNet: 90648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.082794] LNetError: 90648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.092431] LNet: Removed LNI 192.168.203.11@tcp [ 749.101900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.126783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 749.136650] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 749.377624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 749.385186] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 749.709908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.11@tcp [ 749.718119] LNetError: 90919:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 750.176257] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 750.176771] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 750.791302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.11@tcp [ 754.808240] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:50:57 (1786935057) [ 755.411788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg311-server [ 755.951665] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:50:58 (1786935058) [ 756.576450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg311-server [ 757.132139] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:51:00 (1786935060) [ 757.754142] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg311-server [ 758.317330] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:51:01 (1786935061) [ 758.352074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.362661] LNet: 92541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.366671] LNetError: 92541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.369647] LNet: Removed LNI 192.168.203.11@tcp [ 758.375378] LNet: Removed LNI 192.168.203.11@tcp1 [ 758.386771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.413307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.424469] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 758.699198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 758.723546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 758.742140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 758.768513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 758.786315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 758.802885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 760.954141] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:51:03 (1786935063) [ 760.979378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.989187] LNet: 93265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.992513] LNetError: 93265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.995611] LNetError: 93265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.002516] LNet: Removed LNI 192.168.203.11@tcp [ 761.013675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.035482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.045952] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 763.379255] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:51:06 (1786935066) [ 763.407563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.416578] LNet: 93967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.418959] LNetError: 93967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.428345] LNet: Removed LNI 192.168.203.11@tcp [ 763.439934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.462956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.471289] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 765.825976] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:51:08 (1786935068) [ 766.331428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 766.882020] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:51:09 (1786935069) [ 767.027364] LNet: Removed LNI 192.168.203.11@tcp [ 767.255557] Key type .llcrypt unregistered [ 767.256697] Key type ._llcrypt unregistered [ 769.221215] Key type ._llcrypt registered [ 769.222730] Key type .llcrypt registered [ 769.288056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.293922] alg: No test for adler32 (adler32-zlib) [ 770.096893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 770.110904] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 772.964802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 774.475588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.111@tcp [ 774.596344] LNet: 95430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.600665] LNetError: 95430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.611466] LNet: Removed LNI 192.168.203.11@tcp [ 774.763329] Key type .llcrypt unregistered [ 774.764326] Key type ._llcrypt unregistered [ 778.380099] Key type ._llcrypt registered [ 778.381787] Key type .llcrypt registered [ 778.847468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 779.382173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 779.907933] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:51:22 (1786935082) [ 780.428679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 780.963764] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:51:23 (1786935083) [ 781.581822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg311-server [ 782.155847] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:51:25 (1786935085) [ 782.783185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg311-server [ 783.358363] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:51:26 (1786935086) [ 783.845144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.858944] alg: No test for adler32 (adler32-zlib) [ 784.735479] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 784.830359] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 786.416141] Key type lgssc registered [ 786.797625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 794.433242] Key type lgssc unregistered [ 794.549303] LNet: 98601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.552616] LNetError: 98601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.563304] LNet: Removed LNI 192.168.203.11@tcp [ 794.791386] Key type .llcrypt unregistered [ 794.792639] Key type ._llcrypt unregistered [ 799.435284] Key type ._llcrypt registered [ 799.436386] Key type .llcrypt registered [ 799.884791] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:51:42 (1786935102) [ 799.990497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.995390] alg: No test for adler32 (adler32-zlib) [ 800.793371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.814204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 800.830057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 800.848601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 800.866892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 802.913380] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:51:45 (1786935105) [ 802.954743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.966456] LNet: 100010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.980358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.004952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.014284] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 803.334490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.343950] LNetError: 100159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.353296] LNet: Removed LNI 192.168.203.11@tcp [ 803.361824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.381763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 803.392050] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 803.546077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.553775] LNet: 100187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.556085] LNet: 100187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 803.562502] LNet: Removed LNI 192.168.203.11@tcp [ 803.570866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.595064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 803.606417] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 805.626720] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:51:48 (1786935108) [ 805.666967] LNet: 100773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.669566] LNetError: 100773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.671799] LNetError: 100773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 805.680387] LNet: Removed LNI 192.168.203.11@tcp [ 805.891377] Key type .llcrypt unregistered [ 805.892404] Key type ._llcrypt unregistered [ 807.785578] Key type ._llcrypt registered [ 807.786818] Key type .llcrypt registered [ 807.843541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.848550] alg: No test for adler32 (adler32-zlib) [ 808.645184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.668297] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 808.669684] LNet: *** cfs_fail_loc=e003, val=100*** [ 808.891801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 808.901501] Ping reply received with mapping: [ 808.903393] found metadata reply NI[192.168.203.11@tcp] [ 808.904490] 0b cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 808.906100] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 808.907369] 5a 5a 5a 5a 5a 5a 5a 5a [ 808.947990] LNet: 101177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.950947] LNetError: 101177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.959260] LNet: Removed LNI 192.168.203.11@tcp [ 809.171402] Key type .llcrypt unregistered [ 809.172531] Key type ._llcrypt unregistered [ 812.432932] Key type ._llcrypt registered [ 812.434172] Key type .llcrypt registered [ 812.838662] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:51:55 (1786935115) [ 812.998251] Key type .llcrypt unregistered [ 812.999175] Key type ._llcrypt unregistered [ 814.720466] Key type ._llcrypt registered [ 814.721524] Key type .llcrypt registered [ 814.778566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.783295] alg: No test for adler32 (adler32-zlib) [ 815.649688] LNetError: 102232:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 815.790351] Key type .llcrypt unregistered [ 815.791571] Key type ._llcrypt unregistered [ 817.326560] Key type ._llcrypt registered [ 817.327679] Key type .llcrypt registered [ 817.732142] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:52:00 (1786935120) [ 817.896370] Key type .llcrypt unregistered [ 817.897556] Key type ._llcrypt unregistered [ 819.763068] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.765336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.914590] Key type ._llcrypt registered [ 821.915537] Key type .llcrypt registered [ 821.970924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.975855] alg: No test for adler32 (adler32-zlib) [ 822.771148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.788917] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.217500] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:52:09 (1786935129) [ 826.246331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.254387] LNet: 104013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.256978] LNetError: 104013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 826.265679] LNet: Removed LNI 10.1.2.3@tcp [ 826.275736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.489588] Key type .llcrypt unregistered [ 826.490651] Key type ._llcrypt unregistered [ 828.229211] Key type ._llcrypt registered [ 828.230278] Key type .llcrypt registered [ 828.284974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.290325] alg: No test for adler32 (adler32-zlib) [ 829.102430] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 829.126761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 829.179314] LNet: 104310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.181646] LNetError: 104310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.189260] LNet: Removed LNI 192.168.203.11@tcp [ 829.423294] Key type .llcrypt unregistered [ 829.424404] Key type ._llcrypt unregistered [ 833.115627] Key type ._llcrypt registered [ 833.116746] Key type .llcrypt registered [ 833.572230] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:52:16 (1786935136) [ 833.788340] Key type .llcrypt unregistered [ 833.789723] Key type ._llcrypt unregistered [ 835.842945] Key type ._llcrypt registered [ 835.844322] Key type .llcrypt registered [ 835.923237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.930208] alg: No test for adler32 (adler32-zlib) [ 838.698080] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:52:21 (1786935141) [ 838.726696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.743667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.768087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.776338] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 839.003195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.014845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 839.030608] LNet: Removed LNI 192.168.203.11@tcp [ 839.111778] LNet: 106063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.248237] Key type .llcrypt unregistered [ 839.249653] Key type ._llcrypt unregistered [ 842.560149] Key type ._llcrypt registered [ 842.562256] Key type .llcrypt registered [ 842.973308] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:52:25 (1786935145) [ 843.074296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 843.079534] alg: No test for adler32 (adler32-zlib) [ 843.884933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.903746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.911625] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 844.166994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 844.180107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.11@tcp [ 844.187415] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 844.189300] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 844.191291] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 844, 0, 0 [ 844.193730] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 844.194693] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 844.196179] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 844.197498] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 844, 0, 0 [ 844.199357] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 844.200535] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 844.202226] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 844.203608] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 844, 0, 894 [ 844.205647] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 844.206845] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 844.208318] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 844.209709] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 844, 0, 894 [ 844.211991] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 846.046784] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:52:28 (1786935148) [ 846.161456] LNet: 107600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.163687] LNetError: 107600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.169302] LNet: Removed LNI 192.168.203.11@tcp [ 846.358709] Key type .llcrypt unregistered [ 846.359847] Key type ._llcrypt unregistered [ 848.100592] Key type ._llcrypt registered [ 848.102064] Key type .llcrypt registered [ 848.164289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.169260] alg: No test for adler32 (adler32-zlib) [ 848.968702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 848.979652] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 851.830870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 854.611072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.623805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 854.633135] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 854.961216] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.047173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.050929] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.053555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.161770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 857.170483] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 860.557771] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:52:43 (1786935163) [ 860.603164] LNet: 109119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.605974] LNetError: 109119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.608849] LNet: Removed LNI 192.168.203.11@tcp [ 861.664123] LNet: Removed LNI 192.168.203.11@tcp1 [ 861.665813] LNetError: 109119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 861.668312] LNetError: 109119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 861.677450] LNet: Removed LNI 10.1.2.3@tcp99 [ 861.911408] Key type .llcrypt unregistered [ 861.912577] Key type ._llcrypt unregistered [ 864.020150] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.022301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 866.168419] Key type ._llcrypt registered [ 866.169302] Key type .llcrypt registered [ 866.219682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.224245] alg: No test for adler32 (adler32-zlib) [ 867.020114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.038702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.047414] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 867.253576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 867.260479] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 867.865280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 867.879897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 867.893728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 867.900412] LNetError: 109871:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 867.911800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 867.925059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 868.041585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 868.063320] LNet: 109891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.065553] LNetError: 109891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.067727] LNet: Removed LNI 10.1.2.3@tcp [ 869.094326] LNet: Removed LNI 192.168.203.11@tcp [ 869.330391] Key type .llcrypt unregistered [ 869.331482] Key type ._llcrypt unregistered [ 872.974530] Key type ._llcrypt registered [ 872.976327] Key type .llcrypt registered [ 873.442074] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:52:56 (1786935176) [ 873.578150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.584026] alg: No test for adler32 (adler32-zlib) [ 874.381913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.403673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.412672] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 874.642294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 874.670425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg311-client@tcp [ 876.567595] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:52:59 (1786935179) [ 876.595468] LNet: 111419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.598131] LNetError: 111419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.609487] LNet: Removed LNI 192.168.203.11@tcp [ 876.831387] Key type .llcrypt unregistered [ 876.833241] Key type ._llcrypt unregistered [ 878.635910] Key type ._llcrypt registered [ 878.636965] Key type .llcrypt registered [ 878.686712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.693115] alg: No test for adler32 (adler32-zlib) [ 879.481133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 879.488084] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 879.499606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 881.295190] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:53:04 (1786935184) [ 881.736538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 882.214501] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:53:05 (1786935185) [ 882.650507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 883.160661] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:53:06 (1786935186) [ 883.194022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.202539] LNet: 112843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.205706] LNetError: 112843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.215847] LNet: Removed LNI 192.168.203.11@tcp [ 883.224490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.243025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.252994] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 883.484672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 883.499874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 885.510076] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:53:08 (1786935188) [ 885.553372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.562715] LNet: 113553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.565852] LNetError: 113553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.574799] LNet: Removed LNI 192.168.203.11@tcp [ 885.581962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.598693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 885.609637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 885.819289] Key type .llcrypt unregistered [ 885.820480] Key type ._llcrypt unregistered [ 889.122602] Key type ._llcrypt registered [ 889.123894] Key type .llcrypt registered [ 889.549756] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:53:12 (1786935192) [ 889.646379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 889.651545] alg: No test for adler32 (adler32-zlib) [ 890.442885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.458225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.465415] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 890.664478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 890.683274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 892.706655] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:53:15 (1786935195) [ 892.740607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.751838] LNet: 115106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.754694] LNetError: 115106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.762625] LNet: Removed LNI 192.168.203.11@tcp [ 892.772897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.794520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 892.805230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 894.803734] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:53:17 (1786935197) [ 894.836196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.844172] LNet: 115687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.853755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.880399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 895.003965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 895.015277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 895.034437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 895.149739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 895.160726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 895.183238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.195769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.216769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 895.328087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 895.339574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 895.358244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 895.464738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 895.476865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 897.470158] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:53:20 (1786935200) [ 897.506290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.516808] LNet: 116321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.519640] LNet: 116321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 897.531386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.551349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.559804] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 897.788121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 897.795492] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 898.068531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 898.088162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 898.158819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.166129] LNetError: 116603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.168941] LNet: Removed LNI 192.168.203.11@tcp [ 898.179276] LNet: Removed LNI 192.168.203.11@tcp2 [ 898.186443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.210162] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 898.213845] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 898.216821] LNet: 116609:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 898.376423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 898.434657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.488134] LNet: Removed LNI 192.168.203.11@tcp [ 899.489499] LNetError: 116646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.491858] LNetError: 116646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 899.500314] LNet: Removed LNI 192.168.203.11@tcp2 [ 899.508403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.529333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.539639] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 899.783915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 899.791499] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 900.117713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 900.135034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 900.157526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 900.232662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.239416] LNet: 116930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.241877] LNet: 116930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 901.280126] LNet: Removed LNI 192.168.203.11@tcp [ 902.310311] LNet: Removed LNI 192.168.203.11@tcp2 [ 902.319322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.344936] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 902.348813] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 902.352205] LNet: 116937:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.512121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 902.585406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.595730] LNetError: 116974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.598788] LNetError: 116974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 902.602121] LNet: Removed LNI 192.168.203.11@tcp [ 903.654304] LNet: Removed LNI 192.168.203.11@tcp2 [ 903.661860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.677737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.686605] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 903.902764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 903.910924] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 904.173305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 904.191706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 904.262279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.268861] LNet: 117253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.271575] LNet: 117253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 905.312136] LNet: Removed LNI 192.168.203.11@tcp [ 905.320281] LNet: Removed LNI 192.168.203.11@tcp2 [ 905.327496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.349149] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 905.351802] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 905.354709] LNet: 117259:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 905.489682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 905.549871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.592096] LNet: Removed LNI 192.168.203.11@tcp [ 906.599242] LNet: Removed LNI 192.168.203.11@tcp2 [ 906.605934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.621418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.628362] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 906.822308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 906.828057] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 907.093610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 907.107655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 907.126056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 907.199333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.209536] LNetError: 117580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.213016] LNetError: 117580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 907.217121] LNet: Removed LNI 192.168.203.11@tcp [ 908.260327] LNet: Removed LNI 192.168.203.11@tcp2 [ 908.268736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.290770] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 908.293455] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 908.295402] LNet: 117587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 908.446415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 908.515349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.524111] LNet: Removed LNI 192.168.203.11@tcp [ 909.542351] LNet: Removed LNI 192.168.203.11@tcp2 [ 909.551417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.575632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.586534] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 909.867374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 909.876126] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 910.160483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 910.174331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 910.195530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 910.301053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.328125] LNet: Removed LNI 192.168.203.11@tcp [ 911.336283] LNet: Removed LNI 192.168.203.11@tcp2 [ 911.343470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.361528] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 911.363412] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 911.495471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 911.596417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.608127] LNet: Removed LNI 192.168.203.11@tcp [ 912.615215] LNet: Removed LNI 192.168.203.11@tcp2 [ 912.621929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.637858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.645840] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 912.844623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 912.851450] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 913.114465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 913.126425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 913.135708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 913.152965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 913.252789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.258940] LNet: 118279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.261724] LNet: 118279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 914.272124] LNet: Removed LNI 192.168.203.11@tcp [ 915.302248] LNet: Removed LNI 192.168.203.11@tcp2 [ 915.308545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.326588] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 915.328379] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 915.330151] LNet: 118286:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 915.332038] LNet: 118286:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 915.452752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 917.474866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 917.988753] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:53:40 (1786935220) [ 918.022579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.032264] LNetError: 119043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.034305] LNetError: 119043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 918.036955] LNet: Removed LNI 192.168.203.11@tcp [ 919.078443] LNet: Removed LNI 192.168.203.11@tcp2 [ 919.085050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.169455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.179266] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 919.382643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 919.389723] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.632364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.638644] LNet: Removed LNI 10.1.2.3@tcp [ 919.645227] LNet: Removed LNI 192.168.203.11@tcp [ 919.652095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.671860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 919.681872] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 919.683821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.803695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 919.810245] LNet: Removed LNI 192.168.203.11@tcp [ 919.819245] LNet: Removed LNI 10.1.2.3@tcp [ 921.603858] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:53:44 (1786935224) [ 921.642355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.659552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.678485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.686609] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 921.928971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 921.943432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 921.955277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 921.966659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 921.978609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 921.991198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 922.005855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 922.018658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 922.032184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 922.045322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 922.058253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 922.104411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 924.087620] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:53:47 (1786935227) [ 924.132366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.158459] LNet: Removed LNI 192.168.203.11@tcp [ 925.165480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.193921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 925.261751] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 925.264949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 925.267812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 927.359152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.367419] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 927.579795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 927.587316] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 928.196484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 928.210287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 928.232636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.11@tcp -m PUT -e local_timeout -r 1 [ 928.254567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 928.272420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.274125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.353692] LNetError: 120722:0:(lib-move.c:4801:lnet_parse()) 192.168.203.11@tcp, src 192.168.203.11@tcp: Dropping PUT (error -108 looking up sender) [ 929.376319] LNet: 121184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.378623] LNet: 121184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 929.381386] LNet: Removed LNI 10.1.2.3@tcp [ 930.406271] LNet: Removed LNI 192.168.203.11@tcp [ 930.543620] Key type .llcrypt unregistered [ 930.544897] Key type ._llcrypt unregistered [ 933.842845] Key type ._llcrypt registered [ 933.843914] Key type .llcrypt registered [ 934.243093] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:53:57 (1786935237) [ 934.344943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 934.350118] alg: No test for adler32 (adler32-zlib) [ 935.142664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.250163] LNet: 122035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.472319] Key type .llcrypt unregistered [ 935.473780] Key type ._llcrypt unregistered [ 937.351441] Key type ._llcrypt registered [ 937.352503] Key type .llcrypt registered [ 937.409047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.415153] alg: No test for adler32 (adler32-zlib) [ 938.212578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 938.222157] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 940.859591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 940.905730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 941.074990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.111@tcp [ 941.152121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 941.164549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 941.174764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 941.186677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 941.198354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 941.209164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 941.219706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 941.230815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 941.237047] LNet: 122487:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 941.243568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 941.254494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 941.265231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 941.275662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 941.281068] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.111@tcp [ 941.287023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 941.300097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 941.498657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 941.571074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 941.581709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 941.593461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 941.606330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 941.617666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 941.628123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 941.639450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 941.650235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 941.660690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 941.671976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 941.686019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 941.700159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 941.711515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 941.723739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 941.913553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.111@tcp [ 941.990867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.002184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.013933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.025284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 942.036778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 942.049611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 942.063359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 942.076705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 942.083787] LNet: 122721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 942.086074] LNet: 122721:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 942.092050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 942.102559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 942.114129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 942.125070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 942.135512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 942.147567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 942.336373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 942.408630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.419769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.429740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.439543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 942.453542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 942.465756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 942.479167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 942.492530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 942.503756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 942.516583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 942.530610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 942.543365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 942.554300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 942.564927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 942.667583] LNet: 122888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.669700] LNetError: 122888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.678275] LNet: Removed LNI 192.168.203.11@tcp [ 942.810405] Key type .llcrypt unregistered [ 942.811384] Key type ._llcrypt unregistered [ 947.496989] Key type ._llcrypt registered [ 947.498160] Key type .llcrypt registered [ 947.900751] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:54:10 (1786935250) [ 948.001364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.006267] alg: No test for adler32 (adler32-zlib) [ 948.798144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.910214] LNet: 123773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.116279] Key type .llcrypt unregistered [ 949.117531] Key type ._llcrypt unregistered [ 950.982592] Key type ._llcrypt registered [ 950.983678] Key type .llcrypt registered [ 951.045672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.051571] alg: No test for adler32 (adler32-zlib) [ 951.848788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 951.859731] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 954.518109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 957.082464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.096152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 957.104805] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 957.379801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 957.545576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.111@tcp [ 957.615256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.624944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 957.637389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.647750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.658307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.668618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.680697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.692363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.699092] LNet: 124427:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.706969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.718821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 957.732063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 957.746398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 957.758230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 957.769252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 957.955138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 957.985070] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.111@tcp, 192.168.203.111@tcp1 [ 958.031745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.043439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 958.053750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.065504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.077664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.087514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 958.098470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 958.107859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 958.117017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 958.127689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.139248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.150531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.162084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.172577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.346452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.111@tcp [ 958.413634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.423194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 958.434597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.444265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.454234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.464308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 958.473759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 958.482997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 958.489115] LNet: 124661:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 958.491173] LNet: 124661:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 958.497460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 958.511465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.525495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.538901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.551330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.563177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.745695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 958.811698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.821166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 958.829731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.838715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.847123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.856409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 958.865051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 958.873608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 958.881928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 958.891054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.900312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.908707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.918480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.928050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.992687] LNet: 124863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.994790] LNetError: 124863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.997047] LNet: Removed LNI 192.168.203.11@tcp [ 961.004221] LNet: Removed LNI 192.168.203.11@tcp1 [ 961.120432] Key type .llcrypt unregistered [ 961.121378] Key type ._llcrypt unregistered [ 965.943699] Key type ._llcrypt registered [ 965.944795] Key type .llcrypt registered [ 966.326278] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:54:29 (1786935269) [ 966.432159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.437305] alg: No test for adler32 (adler32-zlib) [ 967.231012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.247520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.255420] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 967.462422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 967.470077] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 967.718797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 967.742984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 968.800136] LNet: Removed LNI 192.168.203.11@tcp [ 970.625286] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:54:33 (1786935273) [ 971.065837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 971.564404] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:54:34 (1786935274) [ 973.654669] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:54:36 (1786935276) [ 975.745773] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:54:38 (1786935278) [ 977.824481] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:54:40 (1786935280) [ 979.861418] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:54:42 (1786935282) [ 981.822819] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:54:44 (1786935284) [ 983.761290] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:54:46 (1786935286) [ 985.700546] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:54:48 (1786935288) [ 987.689400] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:54:50 (1786935290) [ 989.577432] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:54:52 (1786935292) [ 991.559709] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:54:54 (1786935294) [ 993.766952] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:54:56 (1786935296) [ 995.843485] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:54:58 (1786935298) [ 997.865202] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:55:00 (1786935300) [ 999.914448] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:55:02 (1786935302) [ 1002.150789] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:55:05 (1786935305) [ 1004.704037] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:55:07 (1786935307) [ 1006.896424] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:55:09 (1786935309) [ 1006.923786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1006.933803] LNet: 136537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.936863] LNetError: 136537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1006.945835] LNet: Removed LNI 192.168.203.11@tcp2 [ 1006.953855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.991954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.000421] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1007.332594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.345387] LNet: Removed LNI 192.168.203.11@tcp [ 1007.350969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.368406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1007.375465] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1009.379134] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:55:12 (1786935312) [ 1011.451387] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:55:14 (1786935314) [ 1013.450355] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:55:16 (1786935316) [ 1015.436524] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:55:18 (1786935318) [ 1017.373798] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:55:20 (1786935320) [ 1019.583566] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:55:22 (1786935322) [ 1021.849705] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:55:24 (1786935324) [ 1023.837718] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:55:26 (1786935326) [ 1025.874786] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:55:28 (1786935328) [ 1027.920486] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:55:30 (1786935330) [ 1029.937889] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:55:32 (1786935332) [ 1031.953919] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:55:34 (1786935334) [ 1034.060899] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 896 sec =========== 22:55:36 (1786935336) [ 1034.145119] LNet: 144136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.148072] LNet: 144136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1034.151597] LNetError: 144136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.154092] LNetError: 144136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1034.166470] LNet: Removed LNI 192.168.203.11@tcp [ 1034.385434] Key type .llcrypt unregistered [ 1034.386455] Key type ._llcrypt unregistered [ 1038.270219] Key type ._llcrypt registered [ 1038.271822] Key type .llcrypt registered [ 1038.415462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.421830] alg: No test for adler32 (adler32-zlib) [ 1039.263210] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1039.342499] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1040.920153] Key type lgssc registered [ 1041.292171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.616156] Lustre: Mounted lustre-client - version 2.17.57_1_g37263e8 [ 1065.973358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1068.094095] Lustre: Unmounted lustre-client [ 1079.705324] Key type lgssc unregistered [ 1079.834402] LNet: 147081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.837111] LNetError: 147081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.843702] LNet: Removed LNI 192.168.203.11@tcp [ 1080.091089] Key type .llcrypt unregistered [ 1080.092391] Key type ._llcrypt unregistered