[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465471526 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003352] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007518] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.014017] *** VALIDATE selinux *** [ 0.021778] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026750] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028045] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032230] *** VALIDATE proc *** [ 0.033218] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042065] debug: unmapping init [mem 0xffffffffa3e59000-0xffffffffa3e60fff] [ 0.044176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045638] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053281] rcu: Hierarchical SRCU implementation. [ 0.055461] smp: Bringing up secondary CPUs ... [ 0.056988] x86: Booting SMP configuration: [ 0.057046] .... node #0, CPUs: #1 #2 #3 [ 0.063203] smp: Brought up 1 node, 4 CPUs [ 0.065024] smpboot: Max logical packages: 1 [ 0.066031] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234024] node 0 deferred pages initialised in 166ms [ 0.238028] devtmpfs: initialized [ 0.239287] x86/mm: Memory block size: 128MB [ 0.243255] gcov: version magic: 0x41383552 [ 0.245883] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247444] pinctrl core: initialized pinctrl subsystem [ 0.248330] [ 0.248977] ************************************************************* [ 0.249026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250024] ** ** [ 0.251021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252085] ** ** [ 0.253020] ** This means that this kernel is built to expose internal ** [ 0.254022] ** IOMMU data structures, which may compromise security on ** [ 0.255026] ** your system. ** [ 0.256020] ** ** [ 0.257026] ** If you see this message and you are not debugging the ** [ 0.258024] ** kernel, report this immediately to your vendor! ** [ 0.259023] ** ** [ 0.260018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261025] ************************************************************* [ 0.262939] NET: Registered protocol family 16 [ 0.263759] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.264088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.265101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.267162] cpuidle: using governor menu [ 0.269757] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272464] PCI: Using configuration type 1 for base access [ 0.274126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.283259] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.284113] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.286148] cryptd: max_cpu_qlen set to 1000 [ 0.288456] ACPI: Added _OSI(Module Device) [ 0.290022] ACPI: Added _OSI(Processor Device) [ 0.292018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.298891] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304262] ACPI: Interpreter enabled [ 0.305086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306022] ACPI: Using IOAPIC for interrupt routing [ 0.307230] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.310394] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.321480] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324117] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331204] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.336437] acpiphp: Slot [2] registered [ 0.337181] acpiphp: Slot [5] registered [ 0.339124] acpiphp: Slot [6] registered [ 0.341110] acpiphp: Slot [3] registered [ 0.342087] acpiphp: Slot [4] registered [ 0.343092] acpiphp: Slot [7] registered [ 0.345093] acpiphp: Slot [8] registered [ 0.346090] acpiphp: Slot [9] registered [ 0.348121] acpiphp: Slot [10] registered [ 0.349065] acpiphp: Slot [11] registered [ 0.350102] acpiphp: Slot [12] registered [ 0.352106] acpiphp: Slot [13] registered [ 0.353150] acpiphp: Slot [14] registered [ 0.355093] acpiphp: Slot [15] registered [ 0.356097] acpiphp: Slot [16] registered [ 0.357085] acpiphp: Slot [17] registered [ 0.358171] acpiphp: Slot [18] registered [ 0.360076] acpiphp: Slot [19] registered [ 0.361108] acpiphp: Slot [20] registered [ 0.362152] acpiphp: Slot [21] registered [ 0.364134] acpiphp: Slot [22] registered [ 0.365094] acpiphp: Slot [23] registered [ 0.367091] acpiphp: Slot [24] registered [ 0.368092] acpiphp: Slot [25] registered [ 0.370093] acpiphp: Slot [26] registered [ 0.371157] acpiphp: Slot [27] registered [ 0.372102] acpiphp: Slot [28] registered [ 0.374136] acpiphp: Slot [29] registered [ 0.375084] acpiphp: Slot [30] registered [ 0.377124] acpiphp: Slot [31] registered [ 0.378075] PCI host bridge to bus 0000:00 [ 0.380021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.382018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.384019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.386016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.388018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.390022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.391166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.393962] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.396282] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.403071] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.406974] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.408020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416512] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.418928] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.421033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.423668] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426822] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.435019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.439019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.443000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.455029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.461023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.476026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.487048] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.519879] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.522401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.525503] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.527634] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.531323] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.535106] iommu: Default domain type: Passthrough [ 0.537539] SCSI subsystem initialized [ 0.539185] ACPI: bus type USB registered [ 0.540135] usbcore: registered new interface driver usbfs [ 0.543128] usbcore: registered new interface driver hub [ 0.545099] usbcore: registered new device driver usb [ 0.547216] pps_core: LinuxPPS API ver. 1 registered [ 0.549026] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.552071] PTP clock support registered [ 0.554204] EDAC MC: Ver: 3.0.0 [ 0.556188] PCI: Using ACPI for IRQ routing [ 0.557559] NetLabel: Initializing [ 0.558013] NetLabel: domain hash size = 128 [ 0.560014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.561106] NetLabel: unlabeled traffic allowed by default [ 0.564278] vgaarb: loaded [ 0.566323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.567025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.574016] clocksource: Switched to clocksource kvm-clock [ 0.687320] VFS: Disk quotas dquot_6.6.0 [ 0.689512] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.692184] *** VALIDATE ramfs *** [ 0.695272] *** VALIDATE hugetlbfs *** [ 0.697182] pnp: PnP ACPI init [ 0.699774] pnp: PnP ACPI: found 6 devices [ 0.720337] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.724406] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.726019] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.727899] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.729829] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.731766] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.734271] NET: Registered protocol family 2 [ 0.736222] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.740056] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.742606] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.746803] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.750699] TCP: Hash tables configured (established 65536 bind 65536) [ 0.753932] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.757670] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760935] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.764524] NET: Registered protocol family 1 [ 0.767250] RPC: Registered named UNIX socket transport module. [ 0.769680] RPC: Registered udp transport module. [ 0.771753] RPC: Registered tcp transport module. [ 0.773694] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.776438] NET: Registered protocol family 44 [ 0.777768] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.779295] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.780806] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.782393] PCI: CLS 0 bytes, default 64 [ 0.783568] Unpacking initramfs... [ 2.190453] debug: unmapping init [mem 0xffff9001bcc64000-0xffff9001bffcffff] [ 2.194090] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.196261] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.199362] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.777735] Initialise system trusted keyrings [ 2.779633] Key type blacklist registered [ 2.781682] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.790711] zbud: loaded [ 2.793757] *** VALIDATE nfs *** [ 2.795197] *** VALIDATE nfs4 *** [ 2.796755] pstore: using deflate compression [ 2.800699] Platform Keyring initialized [ 2.923525] NET: Registered protocol family 38 [ 2.926183] Key type asymmetric registered [ 2.928120] Asymmetric key parser 'x509' registered [ 2.930462] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.933334] io scheduler mq-deadline registered [ 2.934555] io scheduler kyber registered [ 2.936171] io scheduler bfq registered [ 2.938403] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.941533] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.944786] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.948205] ACPI: Power Button [PWRF] [ 2.953900] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.960468] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.968894] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.996361] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.024147] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.028508] Non-volatile memory driver v1.3 [ 3.029706] Linux agpgart interface v0.103 [ 3.063813] virtio_blk virtio1: [vda] 145952 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.066666] vda: detected capacity change from 0 to 74727424 [ 3.083948] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.086693] vdb: detected capacity change from 0 to 1073741824 [ 3.096454] libphy: Fixed MDIO Bus: probed [ 3.105655] usbcore: registered new interface driver usbserial_generic [ 3.107913] usbserial: USB Serial support registered for generic [ 3.110254] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.113568] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.114832] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.117349] mousedev: PS/2 mouse device common for all mice [ 3.120243] rtc_cmos 00:05: RTC can wake from S4 [ 3.124110] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.124458] rtc_cmos 00:05: registered as rtc0 [ 3.131131] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.134791] intel_pstate: CPU model not supported [ 3.138180] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.140367] hid: raw HID events driver (C) Jiri Kosina [ 3.144667] usbcore: registered new interface driver usbhid [ 3.146671] usbhid: USB HID core driver [ 3.147180] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.147994] drop_monitor: Initializing network drop monitor service [ 3.153793] Initializing XFRM netlink socket [ 3.156332] NET: Registered protocol family 10 [ 3.159455] Segment Routing with IPv6 [ 3.160685] NET: Registered protocol family 17 [ 3.162412] mpls_gso: MPLS GSO support [ 3.169112] RAS: Correctable Errors collector initialized. [ 3.170881] AVX version of gcm_enc/dec engaged. [ 3.172542] AES CTR mode by8 optimization enabled [ 3.247902] sched_clock: Marking stable (3247870330, 0)->(4135761497, -887891167) [ 3.252171] registered taskstats version 1 [ 3.254342] Loading compiled-in X.509 certificates [ 3.256477] zswap: loaded using pool lzo/zbud [ 3.279886] Key type big_key registered [ 3.292197] Key type encrypted registered [ 3.293325] ima: No TPM chip found, activating TPM-bypass! [ 3.294905] ima: Allocated hash algorithm: sha1 [ 3.296634] ima: No architecture policies found [ 3.297847] evm: Initialising EVM extended attributes: [ 3.299214] evm: security.selinux [ 3.300380] evm: security.ima [ 3.301142] evm: security.capability [ 3.301974] evm: HMAC attrs: 0x1 [ 3.303710] rtc_cmos 00:05: setting system clock to 2026-08-06 11:45:19 UTC (1786016719) [ 3.309239] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 3.312285] debug: unmapping init [mem 0xffffffffa3b82000-0xffffffffa3e58fff] [ 3.320095] Write protecting the kernel read-only data: 28672k [ 3.322787] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.325175] debug: unmapping init [mem 0xffffffffa2b14000-0xffffffffa2bfffff] [ 3.358844] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.365339] systemd[1]: Detected virtualization kvm. [ 3.367021] systemd[1]: Detected architecture x86-64. [ 3.369020] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.396177] systemd[1]: No hostname configured. [ 3.397954] systemd[1]: Set hostname to . [ 3.400253] random: systemd: uninitialized urandom read (16 bytes read) [ 3.402469] systemd[1]: Initializing machine ID from random generator. [ 3.456555] random: ln: uninitialized urandom read (6 bytes read) [ 3.533454] random: systemd: uninitialized urandom read (16 bytes read) [ 3.535520] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.540629] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.545211] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.202165] device-mapper: uevent: version 1.0.3 [ 4.203958] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 4.880622] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.929207] virtio_net virtio0 ens2: renamed from eth0 [ 4.947697] scsi host0: ata_piix [ 4.963813] scsi host1: ata_piix [ 4.966153] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.968492] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.814026] random: crng init done [ 9.817629] random: 7 urandom warning(s) missed due to ratelimiting [ 9.976987] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.029821] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.423456] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.984754] SELinux: Disabled at runtime. [ 15.110705] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.127280] systemd[1]: Detected virtualization kvm. [ 15.131275] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.775867] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.781776] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.798995] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.809745] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.819390] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.845388] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.867398] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ 17.259300] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 18.298805] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.059802] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.142289] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.708260] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.812216] EDAC sbridge: Ver: 1.1.2 [ 23.194828] Key type dns_resolver registered [ 23.661760] NFS: Registering the id_resolver key type [ 23.665547] Key type id_resolver registered [ 23.667940] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg108-client login: [ 84.623751] libcfs: loading out-of-tree module taints kernel. [ 84.946051] Key type ._llcrypt registered [ 84.951089] Key type .llcrypt registered [ 85.375964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 85.387948] alg: No test for adler32 (adler32-zlib) [ 86.695858] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 87.855111] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 89.719309] Key type lgssc registered [ 90.730862] hrtimer: interrupt took 6869411 ns [ 91.317891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 219.096344] Lustre: Mounted lustre-client [ 223.708274] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.270510] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing check_logdir /tmp/testlogs/ [ 241.539185] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing yml_node [ 244.705578] Lustre: lustre-OST0000-osc-ffff900205462000: disconnect after 23s idle [ 246.569729] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 249.090432] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 251.947361] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 253.843440] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 6 07:49:28 EDT 2026 [ 271.892406] Lustre: Unmounted lustre-client [ 301.690866] Key type lgssc unregistered [ 302.200632] LNet: 8385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.211753] LNetError: 8385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.248146] LNet: Removed LNI 192.168.201.8@tcp [ 303.203175] Key type .llcrypt unregistered [ 303.207573] Key type ._llcrypt unregistered [ 326.616736] Key type ._llcrypt registered [ 326.618991] Key type .llcrypt registered [ 326.754466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 326.764917] alg: No test for adler32 (adler32-zlib) [ 327.634607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 327.678992] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 327.695331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 330.711447] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 330.831307] LNet: 9264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.840861] LNetError: 9264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.855369] LNet: Removed LNI 192.168.201.8@tcp [ 331.409515] Key type .llcrypt unregistered [ 331.411448] Key type ._llcrypt unregistered [ 336.266532] Key type ._llcrypt registered [ 336.268485] Key type .llcrypt registered [ 337.224386] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:50:52 (1786017052) [ 337.520210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.535969] alg: No test for adler32 (adler32-zlib) [ 338.391622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.474975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.494413] LNet: 9680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.517816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.555752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 338.568894] LNet: 9687:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 343.981601] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:50:59 (1786017059) [ 344.210815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.234253] LNet: 10265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.267601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.360806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 350.572454] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:51:05 (1786017065) [ 350.644650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.679381] LNet: 10845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.701639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.837671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 357.626346] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:51:12 (1786017072) [ 357.723275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.752582] LNet: 11429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.776776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.890735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 364.516775] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:51:19 (1786017079) [ 364.641690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.676356] LNet: 12013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.718334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.829667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 365.354451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 371.540440] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:51:26 (1786017086) [ 371.616359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.673522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.748369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.172981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.718829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 379.206440] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:51:34 (1786017094) [ 379.432446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.445678] LNet: 13205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.449627] LNet: 13205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 379.472412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.537450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 386.655617] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:51:41 (1786017101) [ 386.954441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.008372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.097925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 387.136414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 387.494846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 387.549095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 387.849810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 387.881087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 388.191938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 388.249207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 388.506570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 388.542152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 388.895704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 388.924531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.176656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 389.201539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.539872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 389.568354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 389.947304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 390.009305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 390.582736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 390.621904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 390.970516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 391.003798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 391.338967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 391.368681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 391.707400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 391.738994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 397.670563] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:51:52 (1786017112) [ 397.751884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.775824] LNet: 14463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.782862] LNet: 14463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 397.803503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.911099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 397.973576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 403.702839] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:51:58 (1786017118) [ 403.935645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.980495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.051940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 404.086292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 410.615442] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:52:05 (1786017125) [ 410.734996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.793649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.929172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 410.972410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 418.250715] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:52:12 (1786017132) [ 418.376373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.437601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.541504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 418.582591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 425.439400] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:52:20 (1786017140) [ 425.621972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.670214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.750193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 425.811866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 431.567789] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:52:26 (1786017146) [ 431.704506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.736219] LNet: 17397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.742691] LNet: 17397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 431.779323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.898419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 431.927725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 438.733295] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:52:33 (1786017153) [ 438.837544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.881506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.953377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.506141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.556665] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:52:41 (1786017161) [ 446.667245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.723514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.098355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.478669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.148061] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:52:49 (1786017169) [ 454.297679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.352730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.506312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.685867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.853762] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:52:57 (1786017177) [ 462.980678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.047413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.650975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.937960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.555837] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:53:05 (1786017185) [ 470.832279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.867930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.938378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.979634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.490491] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:53:13 (1786017193) [ 478.744218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.834342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.970297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.077696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 486.320209] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:53:21 (1786017201) [ 486.410612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.472230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.571457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 493.409031] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:53:28 (1786017208) [ 493.545423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.617239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.758880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 500.485899] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:53:35 (1786017215) [ 500.581188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.631628] LNet: 22832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.635709] LNet: 22832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 500.663111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.780177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 501.471583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 509.011570] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:53:43 (1786017223) [ 509.224745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.285356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.385601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 509.440714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 517.800878] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:53:52 (1786017232) [ 517.909789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.006598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.077629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 518.131180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 524.920852] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:53:59 (1786017239) [ 525.018186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.063546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.134828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 525.165422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 531.201424] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:54:06 (1786017246) [ 531.428536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.490422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.582674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 531.608260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 537.732689] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:54:12 (1786017252) [ 537.803278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.846752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.898340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 537.968378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 538.010813] LNetError: 25787:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 538.081498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 538.100525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 543.651252] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:54:18 (1786017258) [ 543.805875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.866429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.948977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RjCpKenUt3 [ 544.039324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RjCpKenUt3 [ 544.117896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.RjCpKenUt3 [ 549.601488] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:54:24 (1786017264) [ 549.728181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.778590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.840924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 549.867199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 549.891930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 555.241367] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:54:30 (1786017270) [ 555.367045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.414649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.476892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 555.490182] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 556.655911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 557.814410] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 559.663308] Key type lgssc registered [ 560.609088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 563.828345] LNetError: 27548:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.8@tcp99 has no available nets [ 566.100499] Key type lgssc unregistered [ 566.377755] LNetError: 28656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.401180] LNet: Removed LNI 192.168.201.8@tcp [ 566.865705] Key type .llcrypt unregistered [ 566.867843] Key type ._llcrypt unregistered [ 572.241233] Key type ._llcrypt registered [ 572.242878] Key type .llcrypt registered [ 573.069775] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:54:48 (1786017288) [ 573.426614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 573.438863] alg: No test for adler32 (adler32-zlib) [ 574.331587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.380853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 576.552794] LNet: Message forwarding will be enabled in 2s [ 579.617271] LNet: Message forwarding enabled [ 579.938805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 579.965021] LNet: Message forwarding will stop in 2s to 6s [ 584.436340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 585.419259] LNet: Message forwarding disabled [ 586.934682] LNet: Message forwarding will be enabled in 2s [ 590.559125] LNet: Message forwarding enabled [ 592.883341] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:55:07 (1786017307) [ 592.964148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.992983] LNet: 30056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.345904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.443297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 599.848324] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:55:14 (1786017314) [ 599.985265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.025147] LNet: 30646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.055200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.104433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 600.138497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 600.171133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 600.205386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 600.231962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 600.277631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 600.307630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 600.342797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 600.388639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 600.423371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 600.462525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 600.500879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 600.560296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 600.593539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 600.621850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 600.651191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 600.699735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 600.729920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 600.760689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 600.781396] LNetError: 30689:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 606.916630] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:55:21 (1786017321) [ 607.074687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.123651] LNet: 31255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.179879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.249080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 612.741304] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:55:27 (1786017327) [ 614.276302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg108-server [ 615.692360] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:55:30 (1786017330) [ 615.936739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.955551] LNet: 32191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.980295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.029365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 616.047232] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 616.749850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[9-13]@tcp [ 616.767467] LNet: 32324:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 616.823952] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.10@tcp is being used as a gateway but routing feature is not turned on [ 616.852339] LNetError: 32195:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.10@tcp has gone from up to down [ 617.838909] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.9@tcp is being used as a gateway but routing feature is not turned on [ 617.854190] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 618.877120] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.9@tcp is being used as a gateway but routing feature is not turned on [ 618.906978] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 620.913814] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 620.930551] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 622.899323] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:55:38 (1786017338) [ 624.628406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg108-server [ 624.995636] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.10@tcp is being used as a gateway but routing feature is not turned on [ 625.005704] LNetError: 32195:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 626.195421] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:55:41 (1786017341) [ 626.421460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.452312] LNet: 33255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.462474] LNetError: 33255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.480833] LNet: Removed LNI 192.168.201.8@tcp [ 626.518592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.577108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.608881] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 627.436800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[9-14/2]@tcp [ 627.469948] LNet: 33376:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 627.474857] LNet: 33376:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 627.499647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[9-14/2]@tcp [ 627.515064] LNetError: 33263:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.9@tcp has gone from up to down [ 627.531069] LNetError: 33263:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 634.706979] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:55:49 (1786017349) [ 634.833993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.877039] LNet: 33962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.883960] LNetError: 33962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.929436] LNet: Removed LNI 192.168.201.8@tcp [ 634.975439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.071906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 635.128644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.207652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.319446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.460463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.536931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.665529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 635.758994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.808804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.912669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.059840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.095963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.171958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 636.344367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.426567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.550603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.688753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.740916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.791511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 636.902980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.939207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.000356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.052553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.087362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.174043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 637.308209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.368613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.485524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.605854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.653187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.715893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 637.774243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.807866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.870938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.672904] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:55:58 (1786017358) [ 645.303969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg108-server [ 646.911941] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:56:01 (1786017361) [ 648.312176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg108-server [ 649.991839] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:56:04 (1786017364) [ 650.840153] Key type .llcrypt unregistered [ 650.841912] Key type ._llcrypt unregistered [ 654.808628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 654.884667] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 654.888686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 654.911654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 657.547437] Key type ._llcrypt registered [ 657.549835] Key type .llcrypt registered [ 657.738384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 657.749164] alg: No test for adler32 (adler32-zlib) [ 658.732264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.794651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.827105] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 659.664203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 659.703478] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 661.583464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 661.610687] LNet: Removed LNI 10.1.2.3@tcp [ 661.726881] LNet: 36092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.730118] LNetError: 36092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.748635] LNet: Removed LNI 192.168.201.8@tcp [ 662.384606] Key type .llcrypt unregistered [ 662.391605] Key type ._llcrypt unregistered [ 673.892955] Key type ._llcrypt registered [ 673.899054] Key type .llcrypt registered [ 675.373687] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:56:29 (1786017389) [ 675.946339] Key type .llcrypt unregistered [ 675.948059] Key type ._llcrypt unregistered [ 682.377456] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 682.534220] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 682.540805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 682.544131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 685.070516] Key type ._llcrypt registered [ 685.072832] Key type .llcrypt registered [ 685.254644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.278862] alg: No test for adler32 (adler32-zlib) [ 686.191726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.247968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.276870] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 686.935137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 686.953727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.112561] LNet: 37587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.120124] LNetError: 37587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 689.133039] LNet: Removed LNI 10.1.2.3@tcp [ 689.144791] LNet: Removed LNI 192.168.201.8@tcp [ 689.951571] Key type .llcrypt unregistered [ 689.954500] Key type ._llcrypt unregistered [ 702.021445] Key type ._llcrypt registered [ 702.027319] Key type .llcrypt registered [ 703.859838] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:56:58 (1786017418) [ 704.703337] Key type .llcrypt unregistered [ 704.705507] Key type ._llcrypt unregistered [ 712.273893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.549828] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.568326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.586127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 715.317611] Key type ._llcrypt registered [ 715.321486] Key type .llcrypt registered [ 715.646568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 715.660498] alg: No test for adler32 (adler32-zlib) [ 716.696152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.774635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 716.795550] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 719.802499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 719.841870] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 723.023876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 723.083741] LNet: Removed LNI 10.1.2.3@tcp [ 723.116439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 723.187296] LNet: Removed LNI 10.1.2.31@tcp [ 723.233736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.291579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 724.344639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 724.386032] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 725.813832] LNet: 39509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.820790] LNetError: 39509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 725.836089] LNet: Removed LNI 10.1.2.31@tcp [ 725.845864] LNet: Removed LNI 10.1.2.3@tcp [ 726.897771] Key type .llcrypt unregistered [ 726.900627] Key type ._llcrypt unregistered [ 739.925687] Key type ._llcrypt registered [ 739.926946] Key type .llcrypt registered [ 741.287832] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:57:36 (1786017456) [ 741.871446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.911149] alg: No test for adler32 (adler32-zlib) [ 742.850712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.901237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.918369] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 743.696430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.712854] LNet: 40464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.723860] LNetError: 40464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.743853] LNet: Removed LNI 192.168.201.8@tcp [ 743.778310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.838804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.8@tcp [ 743.862968] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 744.572872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.8@tcp [ 744.599168] LNet: Removed LNI 192.168.201.8@tcp [ 744.685024] LNet: 40599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.297527] Key type .llcrypt unregistered [ 745.300045] Key type ._llcrypt unregistered [ 756.394334] Key type ._llcrypt registered [ 756.396359] Key type .llcrypt registered [ 757.554534] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:57:52 (1786017472) [ 759.296664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg108-server [ 760.589422] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:57:55 (1786017475) [ 761.115520] Key type .llcrypt unregistered [ 761.117127] Key type ._llcrypt unregistered [ 767.270865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 767.414740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 767.428910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 767.434578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 770.015648] Key type ._llcrypt registered [ 770.019406] Key type .llcrypt registered [ 770.194067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.206379] alg: No test for adler32 (adler32-zlib) [ 771.143769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.218136] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 771.957023] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 773.864823] LNet: 42436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.869129] LNetError: 42436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.873935] LNet: Removed LNI 192.168.201.8@tcp1 [ 773.886379] LNet: Removed LNI 10.1.2.3@tcp2 [ 774.394751] Key type .llcrypt unregistered [ 774.396615] Key type ._llcrypt unregistered [ 783.559161] Key type ._llcrypt registered [ 783.560766] Key type .llcrypt registered [ 784.606125] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:58:19 (1786017499) [ 785.085623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 785.100268] alg: No test for adler32 (adler32-zlib) [ 786.027404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.081961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.103831] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 786.839723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.853703] LNet: 43405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.860742] LNetError: 43405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.873743] LNet: Removed LNI 192.168.201.8@tcp [ 786.905273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.958948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 786.978095] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 792.404887] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:58:27 (1786017507) [ 792.533559] LNet: 43992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.540475] LNetError: 43992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.564321] LNet: Removed LNI 192.168.201.8@tcp [ 793.051529] Key type .llcrypt unregistered [ 793.053944] Key type ._llcrypt unregistered [ 797.480984] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 797.563925] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 797.581152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 797.592979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 800.165802] Key type ._llcrypt registered [ 800.168011] Key type .llcrypt registered [ 800.277683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.295691] alg: No test for adler32 (adler32-zlib) [ 801.244705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.313510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.344906] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 802.248148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 802.269039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 804.293726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.336810] LNet: 44739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.342704] LNetError: 44739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 804.349081] LNet: Removed LNI 10.1.2.3@tcp [ 804.357623] LNet: Removed LNI 192.168.201.8@tcp [ 804.381464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.440358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 804.476894] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 804.487149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.025343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.053539] LNet: 44766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.060412] LNetError: 44766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 805.065660] LNetError: 44766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 805.069799] LNet: Removed LNI 10.1.2.3@tcp [ 805.082690] LNet: Removed LNI 192.168.201.8@tcp [ 805.104103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.143474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 805.171045] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 805.173923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.458523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 805.477661] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 806.305914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.324701] LNet: 44928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.328462] LNetError: 44928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 806.331601] LNetError: 44928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 806.335750] LNet: Removed LNI 10.1.2.3@tcp [ 806.337785] LNet: Removed LNI 192.168.201.8@tcp [ 806.347642] LNet: Removed LNI 192.168.201.8@tcp2 [ 806.362660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.406987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 806.423811] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 806.426684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 806.430580] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 812.108124] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:58:47 (1786017527) [ 812.167142] LNet: 45515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.176327] LNetError: 45515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 812.179603] LNetError: 45515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 812.185095] LNet: Removed LNI 10.1.2.3@tcp [ 812.186787] LNet: Removed LNI 192.168.201.8@tcp [ 812.211505] LNet: Removed LNI 192.168.201.8@tcp2 [ 812.685954] Key type .llcrypt unregistered [ 812.689984] Key type ._llcrypt unregistered [ 817.924297] Key type ._llcrypt registered [ 817.932398] Key type .llcrypt registered [ 818.173238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.191222] alg: No test for adler32 (adler32-zlib) [ 819.089540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.146144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.163562] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 819.763829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.779208] LNet: 45919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.783489] LNetError: 45919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.796792] LNet: Removed LNI 192.168.201.8@tcp [ 819.811547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.860586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.880712] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 820.214293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.242882] LNet: Removed LNI 192.168.201.8@tcp [ 820.256777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.347206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.372495] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 820.750055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.767918] LNet: 45979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.772783] LNet: 45979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 820.780722] LNetError: 45979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.785909] LNetError: 45979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 820.799974] LNet: Removed LNI 192.168.201.8@tcp [ 820.816187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.877590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.901132] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 821.282277] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.521421] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.538892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.546315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.933802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.953600] LNet: 46059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.962430] LNetError: 46059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.982702] LNet: Removed LNI 192.168.201.8@tcp [ 823.996421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.040975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 824.057620] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 829.420745] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:59:04 (1786017544) [ 829.506217] LNet: 46650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.513820] LNetError: 46650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.531775] LNet: Removed LNI 192.168.201.8@tcp [ 829.984985] Key type .llcrypt unregistered [ 829.986494] Key type ._llcrypt unregistered [ 834.812639] Key type ._llcrypt registered [ 834.814286] Key type .llcrypt registered [ 835.025158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.043640] alg: No test for adler32 (adler32-zlib) [ 835.929924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.986837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.013948] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 836.758228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.780491] LNet: 47055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.786656] LNetError: 47055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.801664] LNet: Removed LNI 192.168.201.8@tcp [ 836.825394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.904625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 836.938182] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 841.713423] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:59:16 (1786017556) [ 841.943351] LNet: 47647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.948909] LNetError: 47647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.964468] LNet: Removed LNI 192.168.201.8@tcp [ 842.429738] Key type .llcrypt unregistered [ 842.432394] Key type ._llcrypt unregistered [ 847.838463] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 847.937747] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 847.942369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 847.948311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 850.360678] Key type ._llcrypt registered [ 850.363516] Key type .llcrypt registered [ 850.520976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.542605] alg: No test for adler32 (adler32-zlib) [ 851.429686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.466839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.486365] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 852.198810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.219750] LNet: 48092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.225247] LNetError: 48092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.237042] LNet: Removed LNI 192.168.201.8@tcp [ 852.251666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.322857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 852.348811] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 857.840093] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:59:32 (1786017572) [ 857.884471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.903807] LNet: 48681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.909759] LNetError: 48681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.928164] LNet: Removed LNI 192.168.201.8@tcp [ 857.946769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.046517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 862.750320] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:59:37 (1786017577) [ 862.846812] LNet: 49267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.309690] Key type .llcrypt unregistered [ 863.313096] Key type ._llcrypt unregistered [ 868.469496] Key type ._llcrypt registered [ 868.472067] Key type .llcrypt registered [ 868.699144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.708281] alg: No test for adler32 (adler32-zlib) [ 869.628859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.672384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.690937] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 870.385072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.401415] LNet: 49669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.406374] LNetError: 49669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.427156] LNet: Removed LNI 192.168.201.8@tcp [ 870.466393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.557868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 870.580496] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 877.074427] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:59:51 (1786017591) [ 877.163557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.184210] LNet: 50255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.195593] LNetError: 50255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.219502] LNet: Removed LNI 192.168.201.8@tcp [ 877.262019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.318587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 882.920944] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:59:57 (1786017597) [ 884.163255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 885.761661] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:00:00 (1786017600) [ 887.402537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 888.921659] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:00:03 (1786017603) [ 890.184427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 891.733224] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:00:06 (1786017606) [ 892.896637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 894.639903] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:00:09 (1786017609) [ 896.065393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 897.585682] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:00:12 (1786017612) [ 897.755215] LNet: 52313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.342589] Key type .llcrypt unregistered [ 898.346446] Key type ._llcrypt unregistered [ 902.519257] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 902.580262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 902.584782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 902.588841] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 905.122476] Key type ._llcrypt registered [ 905.125336] Key type .llcrypt registered [ 905.264562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.281535] alg: No test for adler32 (adler32-zlib) [ 906.180371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.287312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.320743] LNet: Added LNI 192.168.201.8@tcp [8/512/0/180] [ 906.333933] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 906.335870] LNet: 52638:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 906.475353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.500917] LNet: 52657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.505958] LNetError: 52657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.512796] LNet: Removed LNI 10.1.2.3@tcp [ 906.515874] LNetError: 52657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.534299] LNet: Removed LNI 192.168.201.8@tcp [ 906.562433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.604364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.620397] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 906.669403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.683713] LNet: 52681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.687605] LNetError: 52681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.699574] LNet: Removed LNI 192.168.201.8@tcp [ 906.716647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.768211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.785287] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 906.790468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.792319] LNet: 52689:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 906.866705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.884714] LNet: 52707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.890348] LNetError: 52707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.892768] LNet: Removed LNI 10.1.2.3@tcp [ 906.894881] LNetError: 52707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.907876] LNet: Removed LNI 192.168.201.8@tcp [ 906.923393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.976772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.996229] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 907.002511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.006148] LNet: 52715:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 907.151817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.163907] LNet: 52741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.167777] LNetError: 52741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.171948] LNet: Removed LNI 10.1.2.3@tcp [ 907.173479] LNetError: 52741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.182650] LNet: Removed LNI 192.168.201.8@tcp [ 907.195767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.238862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 907.253709] LNet: Added LNI 192.168.201.8@tcp [8/256/0/182] [ 907.259700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 907.330217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.345323] LNet: 52769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.350335] LNetError: 52769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.354831] LNet: Removed LNI 10.1.2.3@tcp [ 907.356888] LNetError: 52769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.365754] LNet: Removed LNI 192.168.201.8@tcp [ 907.377637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.428358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 907.450058] LNet: Added LNI 192.168.201.8@tcp [8/256/0/182] [ 907.455693] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 907.459942] LNet: 52777:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 907.510905] LNet: Removed LNI 10.1.2.3@tcp [ 907.535042] LNet: Removed LNI 192.168.201.8@tcp [ 908.039328] Key type .llcrypt unregistered [ 908.043755] Key type ._llcrypt unregistered [ 917.484236] Key type ._llcrypt registered [ 917.485720] Key type .llcrypt registered [ 918.497835] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:00:33 (1786017633) [ 918.865671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.884454] alg: No test for adler32 (adler32-zlib) [ 919.738632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.789968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.817332] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 920.399474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.8 [ 920.420709] LNet: 53752:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 920.428437] LNet: Removed LNI 192.168.201.8@tcp [ 926.058596] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:00:40 (1786017640) [ 926.309856] LNet: 54316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.327306] LNetError: 54316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.354774] LNet: Removed LNI 192.168.201.8@tcp [ 927.198610] Key type .llcrypt unregistered [ 927.201086] Key type ._llcrypt unregistered [ 933.614831] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 933.995717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 936.962574] Key type ._llcrypt registered [ 936.968777] Key type .llcrypt registered [ 937.210867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.245573] alg: No test for adler32 (adler32-zlib) [ 938.925213] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.128722] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:01:01 (1786017661) [ 947.351299] LNet: 55249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.381095] LNetError: 55249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.418593] LNet: Removed LNI 10.1.2.3@tcp [ 948.175464] Key type .llcrypt unregistered [ 948.178352] Key type ._llcrypt unregistered [ 954.770388] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 954.923152] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 954.928373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 954.934464] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 957.714386] Key type ._llcrypt registered [ 957.716893] Key type .llcrypt registered [ 957.867962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.884819] alg: No test for adler32 (adler32-zlib) [ 958.947985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.531780] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:01:20 (1786017680) [ 966.996848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 968.501620] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:01:23 (1786017683) [ 968.631631] LNet: 56432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.637757] LNetError: 56432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 968.651873] LNet: Removed LNI 10.1.2.3@tcp [ 969.267202] Key type .llcrypt unregistered [ 969.269691] Key type ._llcrypt unregistered [ 974.839473] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 975.018676] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 975.024347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 975.032583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 977.756640] Key type ._llcrypt registered [ 977.762986] Key type .llcrypt registered [ 977.886450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.903894] alg: No test for adler32 (adler32-zlib) [ 978.817584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.877423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 986.444534] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:01:41 (1786017701) [ 986.847859] LNet: 57664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.852686] LNetError: 57664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 986.863157] LNet: Removed LNI 10.1.2.3@tcp [ 987.544230] Key type .llcrypt unregistered [ 987.547674] Key type ._llcrypt unregistered [ 993.490318] Key type ._llcrypt registered [ 993.495856] Key type .llcrypt registered [ 993.707238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.723714] alg: No test for adler32 (adler32-zlib) [ 994.806755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 994.831205] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1001.201918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1001.669318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_interrupt -r 1 [ 1001.712288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_interrupt -r 1 [ 1001.738516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1001.756583] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1001.939864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1001.962658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1001.985561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.005583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.015295] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1002.033646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.055753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.078478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.098659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1002.110529] LNet: 58102:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1002.124847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1002.145677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1002.167687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.192330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.213293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.236103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.264847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1002.643657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_dropped -r 1 [ 1002.675452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_dropped -r 1 [ 1002.699676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1002.712764] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1002.715012] LNet: Skipped 1 previous similar message [ 1002.893812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.913839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1002.939252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.958229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.979308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1003.002819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1003.027959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.051185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.064135] LNet: 58211:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.081808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.103512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.128971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.150557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1003.173255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.206358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.229411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1003.642994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_aborted -r 1 [ 1003.675992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_aborted -r 1 [ 1003.706769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1003.727553] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1003.916669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.944414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1003.967650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.991619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.009961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.032145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1004.057686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1004.082869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1004.099636] LNet: 58321:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1004.113735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1004.140575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1004.166441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1004.184159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1004.205289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1004.226683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1004.244921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1004.601602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_no_route -r 1 [ 1004.642500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_no_route -r 1 [ 1004.677950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1004.869623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.891555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1004.909238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1004.928150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.950509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.970972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1004.993558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1005.017710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.046263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.077656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1005.108361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.136348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.170147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.192413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.216038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1005.670713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_timeout -r 1 [ 1005.715572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_timeout -r 1 [ 1005.742492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1005.764407] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1005.770910] LNet: Skipped 2 previous similar messages [ 1005.947634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1005.974594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.002843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.029662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.055021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.087464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1006.116548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.142431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.155990] LNet: 58539:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1006.160380] LNet: 58539:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1006.173344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1006.198664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.241628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1006.264568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1006.284882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.311043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.336231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1006.728996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_error -r 1 [ 1006.755900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_error -r 1 [ 1006.779229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1006.949767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.974858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.999153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1007.031983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1007.059178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.088278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1007.111624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1007.148138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1007.174216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1007.201507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1007.230900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1007.261250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1007.292242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.315482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1007.333359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1007.526665] LNet: 58689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.529654] LNetError: 58689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.538686] LNet: Removed LNI 192.168.201.8@tcp [ 1007.819654] Key type .llcrypt unregistered [ 1007.821421] Key type ._llcrypt unregistered [ 1019.493371] Key type ._llcrypt registered [ 1019.495647] Key type .llcrypt registered [ 1020.445554] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:02:15 (1786017735) [ 1021.199282] Key type .llcrypt unregistered [ 1021.201159] Key type ._llcrypt unregistered [ 1025.476879] Key type ._llcrypt registered [ 1025.478858] Key type .llcrypt registered [ 1025.615755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.640796] alg: No test for adler32 (adler32-zlib) [ 1026.516684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1026.540824] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1033.316648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1042.507271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.563083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1042.592427] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1043.666744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_interrupt -r 1 [ 1043.717696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_interrupt -r 1 [ 1043.778515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_interrupt -r 1 [ 1043.837320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_interrupt -r 1 [ 1043.897703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_interrupt -r 1 [ 1043.947415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_interrupt -r 1 [ 1043.984527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_interrupt -r 1 [ 1044.015864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_interrupt -r 1 [ 1044.052341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1044.077216] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1044.085449] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1044.295985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.322067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1044.348490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.377798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.416399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.442840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.488815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1044.529526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1044.546396] LNet: 60156:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1044.568466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1044.599132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.637320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.673843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.714364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.751975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.782745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1045.199748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_dropped -r 1 [ 1045.227958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_dropped -r 1 [ 1045.253534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_dropped -r 1 [ 1045.280467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_dropped -r 1 [ 1045.312950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_dropped -r 1 [ 1045.343972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_dropped -r 1 [ 1045.372513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_dropped -r 1 [ 1045.399616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_dropped -r 1 [ 1045.436883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1045.451990] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1045.455646] LNet: Skipped 2 previous similar messages [ 1045.666474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.686522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1045.712494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.733889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.756225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.780965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.802266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.824061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.835710] LNet: 60277:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1045.848097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.877923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.904317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.936684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.972733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.010827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.050238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1046.459809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_aborted -r 1 [ 1046.495922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_aborted -r 1 [ 1046.534059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_aborted -r 1 [ 1046.560803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_aborted -r 1 [ 1046.589715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_aborted -r 1 [ 1046.619878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_aborted -r 1 [ 1046.652856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_aborted -r 1 [ 1046.696359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_aborted -r 1 [ 1046.729599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1046.743051] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1046.746883] LNet: Skipped 2 previous similar messages [ 1046.896802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.916515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.938590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.963408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.988579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.017975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.041293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.067795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.083599] LNet: 60399:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.099879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.126209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.185759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.253266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.297884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.339818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.367282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1047.925866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_no_route -r 1 [ 1047.958226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_no_route -r 1 [ 1047.980358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_no_route -r 1 [ 1048.004192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_no_route -r 1 [ 1048.030164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_no_route -r 1 [ 1048.073297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_no_route -r 1 [ 1048.133811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_no_route -r 1 [ 1048.177777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_no_route -r 1 [ 1048.206626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1048.409722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.433220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.458369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.496443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.536329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.582597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.614504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.666589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.702492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.751871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.817737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.874112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.921321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.960625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.016414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1049.615548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_timeout -r 1 [ 1049.650405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_timeout -r 1 [ 1049.691486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_timeout -r 1 [ 1049.733923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_timeout -r 1 [ 1049.792564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_timeout -r 1 [ 1049.866281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_timeout -r 1 [ 1049.906165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_timeout -r 1 [ 1049.972855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_timeout -r 1 [ 1050.031699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1050.065517] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1050.072445] LNet: Skipped 5 previous similar messages [ 1050.293463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.323661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.350395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.381902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.418620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.455315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.482599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.527626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.551866] LNet: 60642:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.561355] LNet: 60642:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1050.608099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.669361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.736932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.784334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.843438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.894806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.946956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1051.489333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_error -r 1 [ 1051.518503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_error -r 1 [ 1051.553954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_error -r 1 [ 1051.580965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_error -r 1 [ 1051.611331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_error -r 1 [ 1051.640932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_error -r 1 [ 1051.687739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_error -r 1 [ 1051.733434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_error -r 1 [ 1051.809768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1052.120864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.165310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.226256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.302446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.370960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.421827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.462146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.502740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.537522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.590528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.642441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.689690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.729395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.766922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.798730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1058.607339] LNet: 60840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.612842] LNetError: 60840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.620436] LNet: Removed LNI 192.168.201.8@tcp [ 1059.714663] LNet: Removed LNI 192.168.201.8@tcp1 [ 1060.482796] Key type .llcrypt unregistered [ 1060.491248] Key type ._llcrypt unregistered [ 1080.489709] Key type ._llcrypt registered [ 1080.494460] Key type .llcrypt registered [ 1082.067995] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:03:16 (1786017796) [ 1082.838361] Key type .llcrypt unregistered [ 1082.839919] Key type ._llcrypt unregistered [ 1089.237364] Key type ._llcrypt registered [ 1089.239438] Key type .llcrypt registered [ 1089.350503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.367666] alg: No test for adler32 (adler32-zlib) [ 1090.258551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1090.278950] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1097.221420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1097.909952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_dropped -r 1 [ 1097.951262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_dropped -r 1 [ 1098.019562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1098.044835] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1098.261808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.280734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.323674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.346742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.375361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.411759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.434856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.460397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.474806] LNet: 62091:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.488561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.513746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.537264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.568959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.597125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.623548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.653382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.035183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_error -r 1 [ 1099.063913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_error -r 1 [ 1099.098536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1099.116629] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1099.352433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.386856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.418681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.471404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.510576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.544697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.575950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.609851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.649818] LNet: 62200:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1099.673292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.706574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.735780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.758829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.792879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.822382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.854950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.423406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_timeout -r 1 [ 1100.450712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_timeout -r 1 [ 1100.477262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1100.491170] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1100.697565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.723315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.746972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.773309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.796556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.818580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.843699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.873546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.886968] LNet: 62310:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.901547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.923932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.947545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.972866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.000643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.052877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.105909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1101.371945] LNet: 62351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.382786] LNetError: 62351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1101.399767] LNet: Removed LNI 192.168.201.8@tcp [ 1101.782486] Key type .llcrypt unregistered [ 1101.785066] Key type ._llcrypt unregistered [ 1115.352202] Key type ._llcrypt registered [ 1115.353912] Key type .llcrypt registered [ 1116.678836] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:03:51 (1786017831) [ 1117.268430] Key type .llcrypt unregistered [ 1117.272546] Key type ._llcrypt unregistered [ 1122.939313] Key type ._llcrypt registered [ 1122.943290] Key type .llcrypt registered [ 1123.141882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.158357] alg: No test for adler32 (adler32-zlib) [ 1124.052210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.072384] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1130.353538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1137.869332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1137.924459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1137.936103] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1139.111314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_dropped -r 1 [ 1139.158750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e remote_dropped -r 1 [ 1139.184708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_dropped -r 1 [ 1139.214606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e remote_dropped -r 1 [ 1139.249695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e remote_dropped -r 1 [ 1139.291995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e remote_dropped -r 1 [ 1139.340255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e remote_dropped -r 1 [ 1139.375486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e remote_dropped -r 1 [ 1139.446684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1139.466942] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1139.473390] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.108@tcp [ 1139.656404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.678353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.709543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.732878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.775324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.797525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.834888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.864976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.883775] LNet: 63817:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.896833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.933859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.965722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.018934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.053765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.088522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.124616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.143354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.464751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_error -r 1 [ 1140.497588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e remote_error -r 1 [ 1140.536755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_error -r 1 [ 1140.565514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e remote_error -r 1 [ 1140.597746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e remote_error -r 1 [ 1140.635849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e remote_error -r 1 [ 1140.661895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e remote_error -r 1 [ 1140.687187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e remote_error -r 1 [ 1140.722243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1140.739638] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1140.745729] LNet: Skipped 2 previous similar messages [ 1140.972368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.998046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1141.038218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.062990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.089526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.111731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.139852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.171971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.197259] LNet: 63940:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.222223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.270211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.295415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.325408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.372409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.415446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.452970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.509063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.925566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_timeout -r 1 [ 1141.970525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e remote_timeout -r 1 [ 1142.007201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_timeout -r 1 [ 1142.043892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e remote_timeout -r 1 [ 1142.076985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e remote_timeout -r 1 [ 1142.109899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e remote_timeout -r 1 [ 1142.136647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e remote_timeout -r 1 [ 1142.160639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e remote_timeout -r 1 [ 1142.189801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.108@tcp [ 1142.206342] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1142.420241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.447811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1142.475642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.509958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.543813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.572775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.603583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.635576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.652929] LNet: 64063:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.673234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.710368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.750237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.784274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.810920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.839316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.874179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1142.900629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.677778] LNet: 64141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.682843] LNetError: 64141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.688182] LNet: Removed LNI 192.168.201.8@tcp [ 1147.695883] LNet: Removed LNI 192.168.201.8@tcp1 [ 1147.981586] Key type .llcrypt unregistered [ 1147.983265] Key type ._llcrypt unregistered [ 1161.830955] Key type ._llcrypt registered [ 1161.833310] Key type .llcrypt registered [ 1162.719822] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:04:37 (1786017877) [ 1163.949696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1165.642163] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:04:40 (1786017880) [ 1166.233164] Key type .llcrypt unregistered [ 1166.237150] Key type ._llcrypt unregistered [ 1171.529580] Key type ._llcrypt registered [ 1171.532383] Key type .llcrypt registered [ 1171.763727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.786412] alg: No test for adler32 (adler32-zlib) [ 1172.649710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.675805] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1180.666826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1181.505425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e network_timeout -r 1 [ 1181.549023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e network_timeout -r 1 [ 1181.585828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1181.612136] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1181.912509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.920107] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1181.951556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.990521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.028873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.088578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.108461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.133425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.157783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.175427] LNet: 65684:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.223631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.305851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.355721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.408211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.443409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.487080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.896339] LNet: 65724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.903666] LNetError: 65724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.914397] LNet: Removed LNI 192.168.201.8@tcp [ 1183.333416] Key type .llcrypt unregistered [ 1183.336375] Key type ._llcrypt unregistered [ 1199.703946] Key type ._llcrypt registered [ 1199.705543] Key type .llcrypt registered [ 1199.861983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.880673] alg: No test for adler32 (adler32-zlib) [ 1200.796307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.821221] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1208.158771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1216.186898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.263895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1216.300833] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1217.835268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e network_timeout -r 1 [ 1217.878501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e network_timeout -r 1 [ 1217.919438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e network_timeout -r 1 [ 1217.949490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e network_timeout -r 1 [ 1217.970366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e network_timeout -r 1 [ 1218.015660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e network_timeout -r 1 [ 1218.059652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e network_timeout -r 1 [ 1218.118494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e network_timeout -r 1 [ 1218.180566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1218.208868] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1218.272100] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1218.283241] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.108@tcp [ 1218.428512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.456688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.482652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.507197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.535884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.572751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.612925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.636521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.651851] LNet: 66604:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.661901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.680627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.702038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.724255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.752362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.777715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.235151] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1223.580208] LNet: 66677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.591986] LNetError: 66677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.607676] LNet: Removed LNI 192.168.201.8@tcp [ 1223.625388] LNet: Removed LNI 192.168.201.8@tcp1 [ 1224.145835] Key type .llcrypt unregistered [ 1224.152252] Key type ._llcrypt unregistered [ 1241.810372] Key type ._llcrypt registered [ 1241.812789] Key type .llcrypt registered [ 1243.208785] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:05:57 (1786017957) [ 1243.652572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1243.665632] alg: No test for adler32 (adler32-zlib) [ 1244.590473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.635550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1244.668094] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1245.195329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1245.207746] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1245.982904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 1246.054717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1246.097522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1246.125233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1246.158673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1246.190551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1246.212506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.8@tcp [ 1246.226215] LNetError: 67779:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.8@tcp to recovery [ 1246.687375] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1246.688098] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1248.738050] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1249.786653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1252.840201] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1253.449232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1253.490328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.548777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.568148] LNet: 67887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.576203] LNetError: 67887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.623176] LNet: Removed LNI 192.168.201.8@tcp [ 1254.626246] LNetError: 67887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1254.637357] LNet: Removed LNI 192.168.201.8@tcp1 [ 1254.655584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1254.711798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1254.731912] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1255.383837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1255.402651] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1256.269647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 1256.299659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.341748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1256.370094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1256.398566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1256.433369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1256.459522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1256.486887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.8@tcp [ 1256.501378] LNetError: 68159:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.8@tcp to recovery [ 1256.735630] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1260.078851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1262.879542] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1262.885582] LNet: Skipped 1 previous similar message [ 1267.414690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1267.468953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1267.506563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.862839] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:06:31 (1786017991) [ 1277.237905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.268507] LNet: 68867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.280758] LNetError: 68867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.291323] LNet: Removed LNI 192.168.201.8@tcp [ 1277.321538] LNet: Removed LNI 192.168.201.8@tcp1 [ 1277.352393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.424985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1277.446571] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1278.201917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1278.219752] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1279.375620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 1279.464560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1279.519380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1279.575723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1279.645369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1279.676639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1279.702772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.8@tcp --health 0 [ 1279.806405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1280.479429] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.8@tcp [ 1280.479631] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1280.493923] LNet: Skipped 1 previous similar message [ 1295.315988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1295.492905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1295.562380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.620434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.8@tcp --health 500 [ 1295.702702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1302.372750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1302.447296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.470970] LNet: 69359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.480928] LNetError: 69359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.488859] LNetError: 69359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1303.519170] LNet: Removed LNI 192.168.201.8@tcp [ 1303.529738] LNet: Removed LNI 192.168.201.8@tcp1 [ 1303.562469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.634301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1303.667332] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1304.314689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1304.332438] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1305.052316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 1305.095804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.194081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1305.241710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1305.277956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1305.328288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1305.375977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1305.413268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.8@tcp --health 0 [ 1306.655926] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1306.660658] LNet: Skipped 3 previous similar messages [ 1307.936360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1317.186287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1317.276597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1317.313916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1325.057369] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:07:19 (1786018039) [ 1325.440441] LNet: 70407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.447331] LNetError: 70407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.451581] LNetError: 70407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1325.455452] LNet: Removed LNI 192.168.201.8@tcp [ 1325.469369] LNet: Removed LNI 192.168.201.8@tcp1 [ 1325.997127] Key type .llcrypt unregistered [ 1326.000545] Key type ._llcrypt unregistered [ 1330.852157] Key type ._llcrypt registered [ 1330.854043] Key type .llcrypt registered [ 1331.047830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1331.074836] alg: No test for adler32 (adler32-zlib) [ 1331.948116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1331.969040] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1339.230904] Lustre: DEBUG MARKER: Initial discovery [ 1339.285672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.108@tcp [ 1341.352443] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1341.438227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1341.480873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.108@tcp [ 1341.498584] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1341.526925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.597308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.108@tcp -r 1 -m GET -l 3 [ 1341.625100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.108@tcp -r 1 -m GET -e local_error [ 1341.656925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.108@tcp -r 1 -m PUT -l 6 [ 1342.245280] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1342.943163] Lustre: DEBUG MARKER: Force oleg108-client.virtnet to discover 192.168.201.108@tcp (in background) [ 1342.989917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.108@tcp [ 1344.407336] Lustre: DEBUG MARKER: Force oleg108-server to discover 192.168.201.8@tcp [ 1346.081862] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1347.597344] Lustre: DEBUG MARKER: Wait for 71161 [ 1349.271509] Lustre: DEBUG MARKER: Finished wait on 71161 [ 1349.378437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1349.430158] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1349.435848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1349.678469] LNet: 71620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.684866] LNetError: 71620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.758771] LNet: Removed LNI 192.168.201.8@tcp [ 1351.143778] Key type .llcrypt unregistered [ 1351.147509] Key type ._llcrypt unregistered [ 1367.710808] Key type ._llcrypt registered [ 1367.713342] Key type .llcrypt registered [ 1369.453507] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:08:03 (1786018083) [ 1369.955417] Key type .llcrypt unregistered [ 1369.960362] Key type ._llcrypt unregistered [ 1376.882628] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1377.091092] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1377.100232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1377.110930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1379.840861] Key type ._llcrypt registered [ 1379.842914] Key type .llcrypt registered [ 1380.092634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1380.109890] alg: No test for adler32 (adler32-zlib) [ 1381.136450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.246816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.297818] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1382.301967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1382.321581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1392.149555] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:08:26 (1786018106) [ 1392.274262] LNet: 73689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.281213] LNetError: 73689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1392.288564] LNet: Removed LNI 10.1.2.3@tcp [ 1392.309191] LNet: Removed LNI 192.168.201.8@tcp [ 1392.978279] Key type .llcrypt unregistered [ 1392.982323] Key type ._llcrypt unregistered [ 1399.824780] Key type ._llcrypt registered [ 1399.827272] Key type .llcrypt registered [ 1400.039871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1400.051821] alg: No test for adler32 (adler32-zlib) [ 1400.922790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.960848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1400.977721] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1401.526787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1401.544140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1402.489308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.8@tcp [ 1403.558711] LNetError: Unexpected error -22 connecting to 192.168.201.8@tcp at host 192.168.201.8:988 [ 1409.262429] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:08:44 (1786018124) [ 1409.500636] LNet: 74830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.511220] LNetError: 74830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1409.518565] LNet: Removed LNI 10.1.2.3@tcp [ 1409.531493] LNet: Removed LNI 192.168.201.8@tcp [ 1410.133559] Key type .llcrypt unregistered [ 1410.135605] Key type ._llcrypt unregistered [ 1416.714552] Key type ._llcrypt registered [ 1416.717425] Key type .llcrypt registered [ 1417.096499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.124627] alg: No test for adler32 (adler32-zlib) [ 1418.118876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.209235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.247678] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1419.055887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1419.096506] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 1419.895980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.8@tcp1 --nid 192.168.201.8@tcp2 [ 1424.225562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp1 192.168.201.8@tcp2 [ 1424.242925] LNetError: 75665:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.8@tcp1 to 192.168.201.8@tcp2 - Route table may be misconfigured [ 1424.260931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp2 192.168.201.8@tcp1 [ 1430.122836] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:09:05 (1786018145) [ 1430.216686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.242412] LNet: 76232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.248186] LNetError: 76232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.263196] LNet: Removed LNI 192.168.201.8@tcp1 [ 1431.266510] LNetError: 76232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.290632] LNet: Removed LNI 192.168.201.8@tcp2 [ 1431.315909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.418347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.474237] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1432.290623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.324394] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1433.273984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 1433.305081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.8@tcp [ 1433.335653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp1 192.168.201.8@tcp1 [ 1433.362460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.8@tcp -d 192.168.201.8@tcp -e network_timeout [ 1433.387721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -e network_timeout [ 1433.428542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -e network_timeout [ 1433.475954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -e network_timeout [ 1433.514940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1433.552819] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1433.572190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1433.644508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.730800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1434.470672] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1439.718796] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:09:14 (1786018154) [ 1439.865762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.889058] LNet: 77096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.915589] LNetError: 77096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.920943] LNet: Removed LNI 192.168.201.8@tcp [ 1439.956298] LNet: Removed LNI 192.168.201.8@tcp1 [ 1439.993656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.113329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1440.631617] Key type .llcrypt unregistered [ 1440.636454] Key type ._llcrypt unregistered [ 1451.999392] Key type ._llcrypt registered [ 1452.007482] Key type .llcrypt registered [ 1453.588624] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:09:27 (1786018167) [ 1453.969836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1453.989863] alg: No test for adler32 (adler32-zlib) [ 1454.984908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.673956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1458.784508] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:09:33 (1786018173) [ 1458.933286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.976300] LNet: 78228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.010546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.109530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.142236] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1459.877391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1459.905931] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1460.578381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1460.621277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp1 [ 1460.681673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp1 [ 1467.833995] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:09:42 (1786018182) [ 1470.054101] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg108-server [ 1471.794986] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:09:46 (1786018186) [ 1473.862809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg108-server [ 1475.300520] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:09:50 (1786018190) [ 1477.241292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg108-server [ 1479.132413] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:09:53 (1786018193) [ 1480.570866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg108-server [ 1481.763503] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:09:57 (1786018197) [ 1483.086317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg108-server [ 1484.348754] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:09:59 (1786018199) [ 1486.270697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg108-server [ 1487.689560] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:10:02 (1786018202) [ 1489.445862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg108-server [ 1491.086710] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:10:05 (1786018205) [ 1492.846425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg108-server [ 1494.542837] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:10:09 (1786018209) [ 1495.884644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg108-server [ 1498.141955] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:10:12 (1786018212) [ 1498.289532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.352206] LNet: 82104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.371165] LNetError: 82104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.423161] LNet: Removed LNI 192.168.201.8@tcp [ 1499.424714] LNetError: 82104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.455385] LNet: Removed LNI 192.168.201.8@tcp1 [ 1499.504786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.569279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.607794] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1500.323217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1500.398811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1500.618629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.645547] LNet: 82259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.654745] LNetError: 82259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.673672] LNet: Removed LNI 192.168.201.8@tcp [ 1501.703800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.761252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1501.799394] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1502.587922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1502.696930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.717759] LNet: 82404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.721951] LNetError: 82404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.737552] LNet: Removed LNI 192.168.201.8@tcp [ 1502.754789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.798964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.816146] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1503.493757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1503.558645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1503.674450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.744576] LNet: Removed LNI 192.168.201.8@tcp [ 1504.776119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.841868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1504.885654] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1505.610504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1505.840351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.927220] LNet: Removed LNI 192.168.201.8@tcp [ 1506.973410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.074552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.133814] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1507.941399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1508.040710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1508.415949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.439697] LNet: 82844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.447582] LNet: 82844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1508.453196] LNetError: 82844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.457344] LNetError: 82844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1509.479149] LNet: Removed LNI 192.168.201.8@tcp [ 1509.506913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.577722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1509.599919] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1510.360679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1510.646971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.728495] LNet: Removed LNI 192.168.201.8@tcp [ 1511.779624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.888851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.922444] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1512.527315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1512.589784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1512.913041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.027950] LNet: Removed LNI 192.168.201.8@tcp [ 1514.099344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.195851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1514.237989] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1515.099242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1515.244660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.330190] LNet: Removed LNI 192.168.201.8@tcp [ 1516.367513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.438439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.460904] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1516.980419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1517.051475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1517.253822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.271538] LNet: 83430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.275591] LNet: 83430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1517.281569] LNetError: 83430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.287615] LNetError: 83430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1517.298543] LNet: Removed LNI 192.168.201.8@tcp [ 1517.315242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.356939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1517.377054] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1517.969730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1518.208666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.277646] LNet: Removed LNI 192.168.201.8@tcp [ 1519.314732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.382349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.416792] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1520.236885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1520.300603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1520.516520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1521.574067] LNet: Removed LNI 192.168.201.8@tcp [ 1521.632735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.738848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1521.810954] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1522.669045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1525.167576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.248912] LNet: Removed LNI 192.168.201.8@tcp [ 1526.285522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.326951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1526.352927] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1527.004995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1527.080672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.120936] LNet: Removed LNI 192.168.201.8@tcp [ 1527.155587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.208639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.236416] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1528.008545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1528.093248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.130858] LNet: Removed LNI 192.168.201.8@tcp [ 1528.143374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.198975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1528.308812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.341911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.388395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1528.476348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.536366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.584736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1528.701830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.732992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.800932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1528.836475] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1529.645455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.689755] LNet: Removed LNI 192.168.201.8@tcp [ 1529.707816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.795178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1536.642646] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:10:51 (1786018251) [ 1536.767569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.790904] LNet: 84901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.795482] LNet: 84901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1536.806026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.844394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1536.892241] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/180] [ 1537.736369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.764183] LNetError: 85046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.772947] LNetError: 85046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1537.782890] LNet: Removed LNI 192.168.201.8@tcp231 [ 1537.798855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.868181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1537.893131] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/180] [ 1537.898776] LNet: 85054:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1538.324966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1538.360046] LNet: Removed LNI 192.168.201.8@tcp231 [ 1538.379462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1538.408066] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/180] [ 1539.505486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.527640] LNet: Removed LNI 192.168.201.8@tcp231 [ 1539.541596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.580991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1539.598789] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/0] [ 1539.607438] LNet: 85206:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1539.919643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1539.989704] LNet: Removed LNI 192.168.201.8@tcp231 [ 1539.998845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1540.026305] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/0] [ 1541.043648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.086709] LNet: Removed LNI 192.168.201.8@tcp231 [ 1541.118270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.174686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1541.201190] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/60] [ 1541.211845] LNet: 85357:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1541.617302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1541.652530] LNet: Removed LNI 192.168.201.8@tcp231 [ 1541.667260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1541.688296] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/60] [ 1542.554496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.601761] LNet: Removed LNI 192.168.201.8@tcp231 [ 1542.625524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.686769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1542.712153] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/180] [ 1543.242722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1543.286635] LNet: Removed LNI 192.168.201.8@tcp231 [ 1543.299488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1543.338743] LNet: Added LNI 192.168.201.8@tcp231 [8/256/0/180] [ 1551.351068] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:11:06 (1786018266) [ 1551.557080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.605223] LNet: Removed LNI 192.168.201.8@tcp231 [ 1551.648777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.750563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1551.795193] LNet: Added LNI 192.168.201.8@tcp232 [8/256/0/180] [ 1552.511762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1552.555927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1552.592915] LNet: Removed LNI 192.168.201.8@tcp232 [ 1552.606530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1552.644421] LNet: Added LNI 192.168.201.8@tcp232 [8/256/0/180] [ 1553.215877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1553.253590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1553.281895] LNet: Removed LNI 192.168.201.8@tcp232 [ 1553.295966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1553.316872] LNet: Added LNI 192.168.201.8@tcp232 [8/256/0/180] [ 1553.933876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1553.969485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1553.994168] LNet: Removed LNI 192.168.201.8@tcp232 [ 1554.016082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1554.035639] LNet: Added LNI 192.168.201.8@tcp232 [8/256/0/180] [ 1554.793261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1554.841988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1554.890871] LNet: Removed LNI 192.168.201.8@tcp232 [ 1554.913566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1561.861732] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:11:16 (1786018276) [ 1563.315949] Key type .llcrypt unregistered [ 1563.321341] Key type ._llcrypt unregistered [ 1570.045676] Key type ._llcrypt registered [ 1570.050858] Key type .llcrypt registered [ 1570.273290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.298623] alg: No test for adler32 (adler32-zlib) [ 1571.360238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1571.404650] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1579.918123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1588.984491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.023491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1589.042230] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1590.551425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_interrupt -r 2 [ 1590.590764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_interrupt -r 2 [ 1590.636461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_interrupt -r 2 [ 1590.736532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_interrupt -r 2 [ 1590.796362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_interrupt -r 2 [ 1590.849792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_interrupt -r 2 [ 1590.945542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_interrupt -r 2 [ 1591.014344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_interrupt -r 2 [ 1591.098464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.108@tcp [ 1591.125631] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1591.149515] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1591.489105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1591.514254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1591.539827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1591.583757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1591.619555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1591.668567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1591.708284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1591.756429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1591.779532] LNet: 88012:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1591.799881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1591.836930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1591.903415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1591.967594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1592.024708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1592.100888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1592.692417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_dropped -r 2 [ 1592.734164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_dropped -r 2 [ 1592.773392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_dropped -r 2 [ 1592.818687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_dropped -r 2 [ 1592.867469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_dropped -r 2 [ 1592.898380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_dropped -r 2 [ 1592.930977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_dropped -r 2 [ 1592.966883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_dropped -r 2 [ 1593.013186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.108@tcp [ 1593.039802] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1593.387100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1593.439005] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1593.482536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1593.509605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1593.557324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1593.603241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1593.642561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1593.670196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1593.686275] LNet: 88133:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1593.703049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1593.782675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1593.855171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1593.891904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1593.932730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1593.992247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1594.476665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_aborted -r 2 [ 1594.520688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_aborted -r 2 [ 1594.592288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_aborted -r 2 [ 1594.632881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_aborted -r 2 [ 1594.680657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_aborted -r 2 [ 1594.721083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_aborted -r 2 [ 1594.772090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_aborted -r 2 [ 1594.819604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_aborted -r 2 [ 1594.883979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.108@tcp [ 1594.914724] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1595.277925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1595.318825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1595.380650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1595.418251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1595.440910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1595.469731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1595.493778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1595.521260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1595.537978] LNet: 88253:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1595.551651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1595.577150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1595.604422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1595.633218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1595.657625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1595.686733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1596.314415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_no_route -r 2 [ 1596.375812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_no_route -r 2 [ 1596.428436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_no_route -r 2 [ 1596.479395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_no_route -r 2 [ 1596.520950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_no_route -r 2 [ 1596.564830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_no_route -r 2 [ 1596.631725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_no_route -r 2 [ 1596.692441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_no_route -r 2 [ 1596.774817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.108@tcp [ 1597.111660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1597.135321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1597.159548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1597.187564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1597.212906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1597.235842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1597.267342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1597.305643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1597.354365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1597.390344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1597.415976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1597.452356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1597.490331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1597.519948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1598.070516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e local_timeout -r 2 [ 1598.121605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e local_timeout -r 2 [ 1598.173434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e local_timeout -r 2 [ 1598.214943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e local_timeout -r 2 [ 1598.281665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e local_timeout -r 2 [ 1598.319832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e local_timeout -r 2 [ 1598.352383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e local_timeout -r 2 [ 1598.380446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e local_timeout -r 2 [ 1598.402456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.108@tcp [ 1598.421694] LNet: There was an unexpected network error while writing to 192.168.201.108: rc = -22 [ 1598.427250] LNet: Skipped 1 previous similar message [ 1598.751916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1598.792216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1598.843613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1598.886385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1598.927675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1598.969077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1599.000801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1599.037655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1599.056757] LNet: 88494:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1599.064841] LNet: 88494:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1599.085993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1599.114944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1599.146705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1599.198708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1599.231286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1599.254916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1599.770768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp -m GET -e remote_dropped -r 2 [ 1599.819775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp -d 192.168.201.108@tcp1 -m GET -e remote_dropped -r 2 [ 1599.862068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp -m GET -e remote_dropped -r 2 [ 1599.907747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp -d 192.168.201.8@tcp1 -m GET -e remote_dropped -r 2 [ 1599.944269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp -m GET -e remote_dropped -r 2 [ 1599.986241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.108@tcp1 -m GET -e remote_dropped -r 2 [ 1600.024364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp -m GET -e remote_dropped -r 2 [ 1600.063246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.8@tcp1 -d 192.168.201.8@tcp1 -m GET -e remote_dropped -r 2 [ 1600.094944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.8@tcp 192.168.201.108@tcp [ 1600.116568] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.108@tcp [ 1600.359244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1600.407516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1600.447752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1600.477695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1600.515407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1600.552485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1600.591646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1600.630119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1600.664799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1600.703341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1600.739370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1600.775314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1600.825738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1600.887060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1606.239497] LNet: 88688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.245657] LNetError: 88688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.263233] LNet: Removed LNI 192.168.201.8@tcp [ 1607.265527] LNetError: 88688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.279614] LNet: Removed LNI 192.168.201.8@tcp1 [ 1607.684698] Key type .llcrypt unregistered [ 1607.686398] Key type ._llcrypt unregistered [ 1623.887603] Key type ._llcrypt registered [ 1623.890907] Key type .llcrypt registered [ 1625.087686] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:12:19 (1786018339) [ 1625.522460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.537393] alg: No test for adler32 (adler32-zlib) [ 1626.466098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.502872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.517639] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1627.208267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1627.248047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.8@tcp [ 1627.278317] LNetError: 89667:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.8@tcp to recovery [ 1627.434909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.8@tcp [ 1627.527699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1633.499720] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:12:28 (1786018348) [ 1633.720437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.750278] LNet: 90257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.757768] LNetError: 90257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.776905] LNet: Removed LNI 192.168.201.8@tcp [ 1633.796407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.841363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.860237] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1634.523697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1634.545310] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1635.419868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.8@tcp [ 1635.443881] LNetError: 90523:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.8@tcp to recovery [ 1635.872116] LNet: 1 local NIs in recovery (showing 1): 192.168.201.8@tcp [ 1635.877449] LNet: There was an unexpected network error while writing to 192.168.201.8: rc = -22 [ 1636.707905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.8@tcp [ 1644.731203] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:12:39 (1786018359) [ 1646.474002] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg108-server [ 1647.877425] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:12:42 (1786018362) [ 1649.522925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg108-server [ 1651.108727] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:12:46 (1786018366) [ 1652.559488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg108-server [ 1653.764606] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:12:48 (1786018368) [ 1653.946039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.957877] LNet: 92153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.961582] LNetError: 92153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.973292] LNet: Removed LNI 192.168.201.8@tcp [ 1653.987612] LNet: Removed LNI 192.168.201.8@tcp1 [ 1654.034804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.095631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.118435] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1654.544453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1654.578914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1654.613552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1654.645429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1654.672476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1654.706924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1659.901589] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:12:55 (1786018375) [ 1660.183775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.212492] LNet: 92878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.222122] LNetError: 92878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.229235] LNetError: 92878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1660.249646] LNet: Removed LNI 192.168.201.8@tcp [ 1660.268430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.300794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.315187] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1666.171391] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:13:01 (1786018381) [ 1666.418197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.465246] LNet: 93581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.475423] LNetError: 93581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.495135] LNet: Removed LNI 192.168.201.8@tcp [ 1666.510388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.544274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.560748] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1671.999927] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:13:07 (1786018387) [ 1673.275823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1674.530294] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:13:09 (1786018389) [ 1675.087953] LNet: 94616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.096993] LNetError: 94616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.127723] LNet: Removed LNI 192.168.201.8@tcp [ 1675.565036] Key type .llcrypt unregistered [ 1675.568143] Key type ._llcrypt unregistered [ 1680.568656] Key type ._llcrypt registered [ 1680.570812] Key type .llcrypt registered [ 1680.771370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.789830] alg: No test for adler32 (adler32-zlib) [ 1681.704537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1681.747814] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1688.104764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1691.933913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.108@tcp [ 1692.154757] LNet: 95033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.159936] LNetError: 95033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.170660] LNet: Removed LNI 192.168.201.8@tcp [ 1692.521412] Key type .llcrypt unregistered [ 1692.523716] Key type ._llcrypt unregistered [ 1701.896764] Key type ._llcrypt registered [ 1701.901468] Key type .llcrypt registered [ 1703.047672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1704.348714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1705.961728] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:13:40 (1786018420) [ 1707.325352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1708.992969] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:13:43 (1786018423) [ 1710.461062] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg108-server [ 1711.855385] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:13:46 (1786018426) [ 1713.478320] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg108-server [ 1715.211622] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:13:50 (1786018430) [ 1716.503692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.575231] alg: No test for adler32 (adler32-zlib) [ 1717.671464] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 1717.916393] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1719.615261] Key type lgssc registered [ 1720.916928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.395574] Key type lgssc unregistered [ 1741.779481] LNet: 98224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.786077] LNetError: 98224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.799258] LNet: Removed LNI 192.168.201.8@tcp [ 1742.540466] Key type .llcrypt unregistered [ 1742.547628] Key type ._llcrypt unregistered [ 1756.389327] Key type ._llcrypt registered [ 1756.391850] Key type .llcrypt registered [ 1757.574102] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:14:32 (1786018472) [ 1757.884162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.895861] alg: No test for adler32 (adler32-zlib) [ 1758.896308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.939958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1758.976282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1759.026314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1759.107915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1766.080057] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:14:39 (1786018479) [ 1766.167792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.202413] LNet: 99630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.236398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.339865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.377750] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1767.344666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.368769] LNet: 99783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.373569] LNetError: 99783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.396493] LNet: Removed LNI 192.168.201.8@tcp [ 1767.442287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.566037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1767.626336] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1768.113324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.130835] LNetError: 99811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.150476] LNet: Removed LNI 192.168.201.8@tcp [ 1768.173655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.226991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1768.268305] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1775.607311] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:14:50 (1786018490) [ 1775.775084] LNet: 100397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.782766] LNet: 100397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1775.793973] LNetError: 100397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.832267] LNet: Removed LNI 192.168.201.8@tcp [ 1776.694056] Key type .llcrypt unregistered [ 1776.705761] Key type ._llcrypt unregistered [ 1783.977310] Key type ._llcrypt registered [ 1783.980585] Key type .llcrypt registered [ 1784.165602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1784.183375] alg: No test for adler32 (adler32-zlib) [ 1785.143474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.233428] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1785.237949] LNet: *** cfs_fail_loc=e003, val=100*** [ 1785.832661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1785.856534] LNet: *** cfs_fail_loc=e003, val=100*** [ 1785.861972] LNet: Skipped 2 previous similar messages [ 1785.874410] Ping reply received with mapping: [ 1785.876334] found metadata reply NI[192.168.201.8@tcp] [ 1785.878800] 08 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1785.883887] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1785.889277] 5a 5a 5a 5a 5a 5a 5a 5a [ 1786.063992] LNet: 100799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.073203] LNetError: 100799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.115578] LNet: Removed LNI 192.168.201.8@tcp [ 1787.781348] Key type .llcrypt unregistered [ 1787.782694] Key type ._llcrypt unregistered [ 1800.340960] Key type ._llcrypt registered [ 1800.344263] Key type .llcrypt registered [ 1802.377639] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:15:16 (1786018516) [ 1803.010424] Key type .llcrypt unregistered [ 1803.012099] Key type ._llcrypt unregistered [ 1809.148925] Key type ._llcrypt registered [ 1809.154741] Key type .llcrypt registered [ 1809.493262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1809.535641] alg: No test for adler32 (adler32-zlib) [ 1810.702937] LNetError: 101846:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1811.041565] Key type .llcrypt unregistered [ 1811.043981] Key type ._llcrypt unregistered [ 1816.299700] Key type ._llcrypt registered [ 1816.302393] Key type .llcrypt registered [ 1817.482222] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:15:32 (1786018532) [ 1817.920290] Key type .llcrypt unregistered [ 1817.921805] Key type ._llcrypt unregistered [ 1822.730475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1822.733412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1825.256747] Key type ._llcrypt registered [ 1825.262623] Key type .llcrypt registered [ 1825.434833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1825.452389] alg: No test for adler32 (adler32-zlib) [ 1826.322142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.360350] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1834.315203] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:15:49 (1786018549) [ 1834.422990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.445687] LNet: 103623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.456110] LNetError: 103623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1834.475870] LNet: Removed LNI 10.1.2.3@tcp [ 1834.505618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.175063] Key type .llcrypt unregistered [ 1835.177922] Key type ._llcrypt unregistered [ 1841.029704] Key type ._llcrypt registered [ 1841.031508] Key type .llcrypt registered [ 1841.155097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1841.166486] alg: No test for adler32 (adler32-zlib) [ 1842.100329] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1842.175509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1842.392544] LNet: 103914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.398858] LNetError: 103914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.426859] LNet: Removed LNI 192.168.201.8@tcp [ 1842.988680] Key type .llcrypt unregistered [ 1842.992275] Key type ._llcrypt unregistered [ 1854.030856] Key type ._llcrypt registered [ 1854.039129] Key type .llcrypt registered [ 1855.494839] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:16:10 (1786018570) [ 1856.088934] Key type .llcrypt unregistered [ 1856.090554] Key type ._llcrypt unregistered [ 1861.868688] Key type ._llcrypt registered [ 1861.875289] Key type .llcrypt registered [ 1862.043509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1862.052736] alg: No test for adler32 (adler32-zlib) [ 1869.345850] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:16:23 (1786018583) [ 1869.440376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.494605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.657655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.684471] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1870.618793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.662388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1870.703219] LNet: Removed LNI 192.168.201.8@tcp [ 1870.961949] LNet: 105658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.626643] Key type .llcrypt unregistered [ 1871.630960] Key type ._llcrypt unregistered [ 1883.376268] Key type ._llcrypt registered [ 1883.378328] Key type .llcrypt registered [ 1884.667878] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:16:39 (1786018599) [ 1885.143694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.157830] alg: No test for adler32 (adler32-zlib) [ 1886.108689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.178655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1886.227970] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1887.080899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1887.154887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.8@tcp [ 1887.181703] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1887.191675] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1887.198205] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1887, 0, 0 [ 1887.208734] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1887.214093] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1887.220969] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1887.230445] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1887, 0, 0 [ 1887.240686] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1887.247052] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1887.253514] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1887.259558] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1887, 0, 1937 [ 1887.268095] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1887.284241] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1887.288446] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1887.294093] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1887, 0, 1937 [ 1887.302888] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1893.186809] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:16:48 (1786018608) [ 1893.552464] LNet: 107199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.556789] LNetError: 107199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.642804] LNet: Removed LNI 192.168.201.8@tcp [ 1895.772743] Key type .llcrypt unregistered [ 1895.774597] Key type ._llcrypt unregistered [ 1902.118664] Key type ._llcrypt registered [ 1902.123572] Key type .llcrypt registered [ 1902.409665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1902.430762] alg: No test for adler32 (adler32-zlib) [ 1903.433682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1903.468278] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1911.008551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 1920.703053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.772251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1920.790331] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 1921.810785] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1922.015029] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1922.031897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1922.042590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1924.448930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1924.472141] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1935.013579] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:17:29 (1786018649) [ 1935.206249] LNet: 108709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.219818] LNetError: 108709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.226419] LNet: Removed LNI 192.168.201.8@tcp [ 1936.287221] LNet: Removed LNI 192.168.201.8@tcp1 [ 1936.296975] LNetError: 108709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1936.314136] LNetError: 108709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1936.339099] LNet: Removed LNI 10.1.2.3@tcp99 [ 1937.144174] Key type .llcrypt unregistered [ 1937.145927] Key type ._llcrypt unregistered [ 1943.448635] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1943.461574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1945.974386] Key type ._llcrypt registered [ 1945.978617] Key type .llcrypt registered [ 1946.191926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1946.209154] alg: No test for adler32 (adler32-zlib) [ 1947.171911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.246789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.276438] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1947.973764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1947.993448] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1949.593615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1949.643814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1949.682610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1949.698741] LNetError: 109454:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1949.729745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1949.765497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 1950.143600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1950.204564] LNet: 109474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.209762] LNetError: 109474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1950.215734] LNet: Removed LNI 10.1.2.3@tcp [ 1951.284666] LNet: Removed LNI 192.168.201.8@tcp [ 1951.959725] Key type .llcrypt unregistered [ 1951.962894] Key type ._llcrypt unregistered [ 1969.485636] Key type ._llcrypt registered [ 1969.489528] Key type .llcrypt registered [ 1970.974232] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:18:05 (1786018685) [ 1971.535885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1971.548328] alg: No test for adler32 (adler32-zlib) [ 1972.466636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.525821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.547258] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1973.361838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 1973.512315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg108-client@tcp [ 1981.010697] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:18:15 (1786018695) [ 1981.526485] LNet: 111010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.558821] LNetError: 111010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.576525] LNet: Removed LNI 192.168.201.8@tcp [ 1982.388351] Key type .llcrypt unregistered [ 1982.390457] Key type ._llcrypt unregistered [ 1989.627636] Key type ._llcrypt registered [ 1989.630857] Key type .llcrypt registered [ 1989.864237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1989.885533] alg: No test for adler32 (adler32-zlib) [ 1990.832646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1990.863196] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 1990.898450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1997.608965] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:18:32 (1786018712) [ 1999.438574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2001.465383] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:18:36 (1786018716) [ 2003.012337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2004.852636] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:18:39 (1786018719) [ 2004.954884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.991042] LNet: 112425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.000982] LNetError: 112425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.019548] LNet: Removed LNI 192.168.201.8@tcp [ 2005.042144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.110884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.132396] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2005.895167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2006.005575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2012.668316] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:18:47 (1786018727) [ 2012.881715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.903454] LNet: 113139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.908369] LNetError: 113139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.923465] LNet: Removed LNI 192.168.201.8@tcp [ 2012.979750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.039457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2013.083653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2013.586622] Key type .llcrypt unregistered [ 2013.591213] Key type ._llcrypt unregistered [ 2023.203344] Key type ._llcrypt registered [ 2023.205767] Key type .llcrypt registered [ 2024.185871] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:18:59 (1786018739) [ 2024.559274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.569476] alg: No test for adler32 (adler32-zlib) [ 2025.524958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.600470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.620757] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2026.251903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2026.293023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2031.826673] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:19:06 (1786018746) [ 2032.146646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.177817] LNet: 114696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.184365] LNetError: 114696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.202700] LNet: Removed LNI 192.168.201.8@tcp [ 2032.237228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.300806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2032.351703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2038.438386] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:19:13 (1786018753) [ 2038.556276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.599244] LNet: 115267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.632200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.734746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2039.301384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2039.347154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2039.447806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2040.149792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2040.199809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2040.290032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2040.322274] LNet: 115309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.347238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.441702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2041.039251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2041.080273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2041.137474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2041.686107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2041.745450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2047.767775] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:19:22 (1786018762) [ 2048.130585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2048.147849] LNet: 115909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.186179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2048.277600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2048.298512] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2048.964787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2048.978813] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2049.888229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2049.940475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2050.125331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.141897] LNetError: 116189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.149871] LNet: Removed LNI 192.168.201.8@tcp [ 2050.158290] LNet: Removed LNI 192.168.201.8@tcp2 [ 2050.173505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.213588] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2050.218207] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2050.224718] LNet: 116195:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2050.638661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2050.864680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2051.935332] LNet: Removed LNI 192.168.201.8@tcp [ 2051.939242] LNetError: 116232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.947722] LNetError: 116232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2051.978188] LNet: Removed LNI 192.168.201.8@tcp2 [ 2052.032897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.098661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.113188] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2052.766634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2052.780989] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2053.409792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2053.440865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2053.482670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2053.686780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.711797] LNet: 116516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.717374] LNet: 116516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2054.751535] LNet: Removed LNI 192.168.201.8@tcp [ 2054.754131] LNetError: 116516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.760392] LNetError: 116516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2054.776754] LNet: Removed LNI 192.168.201.8@tcp2 [ 2054.806737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.869190] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2054.875103] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2054.882646] LNet: 116523:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2055.215081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2055.356300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2055.367689] LNet: Removed LNI 192.168.201.8@tcp [ 2056.423313] LNet: Removed LNI 192.168.201.8@tcp2 [ 2056.456624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.532200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2056.551764] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2057.280516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2057.296069] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2058.222451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2058.287739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2058.527875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.553579] LNet: Removed LNI 192.168.201.8@tcp [ 2059.560256] LNetError: 116839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.582023] LNetError: 116839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2059.616888] LNet: Removed LNI 192.168.201.8@tcp2 [ 2059.655948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.711251] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2059.717758] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2059.725960] LNet: 116846:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2060.295799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2060.536836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.599271] LNet: Removed LNI 192.168.201.8@tcp [ 2061.608918] LNet: Removed LNI 192.168.201.8@tcp2 [ 2061.621566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.680475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.711658] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2062.234977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2062.251489] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2063.079233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2063.119442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2063.194652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2063.390860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.415416] LNet: 117167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.420529] LNet: 117167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2064.479141] LNet: Removed LNI 192.168.201.8@tcp [ 2065.512464] LNet: Removed LNI 192.168.201.8@tcp2 [ 2065.566925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.674600] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2065.690397] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2065.693841] LNet: 117174:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2066.299615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2066.585441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.617049] LNet: Removed LNI 192.168.201.8@tcp [ 2067.686691] LNet: Removed LNI 192.168.201.8@tcp2 [ 2067.757511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.833600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.867786] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2068.512595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2068.525381] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2069.328539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2069.379673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2069.443497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2069.758832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.776604] LNetError: 117508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.781595] LNetError: 117508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2069.786419] LNet: Removed LNI 192.168.201.8@tcp [ 2069.795183] LNet: Removed LNI 192.168.201.8@tcp2 [ 2069.812156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.851118] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2069.857576] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2069.864171] LNet: 117514:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2070.203479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2070.528380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2071.583476] LNet: Removed LNI 192.168.201.8@tcp [ 2071.601634] LNet: Removed LNI 192.168.201.8@tcp2 [ 2071.629910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.706696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2071.727859] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2072.601546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2072.654144] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2073.693580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2073.735874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2073.765416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2073.830966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2074.184215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2075.231992] LNet: Removed LNI 192.168.201.8@tcp [ 2076.267875] LNet: Removed LNI 192.168.201.8@tcp2 [ 2076.327550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.444981] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2076.458443] LNet: Added LNI 192.168.201.8@tcp2 [8/256/0/180] [ 2077.033154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2084.047432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2085.545920] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:20:00 (1786018800) [ 2085.661333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.706140] LNet: 118628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.715017] LNet: 118628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2085.732684] LNet: Removed LNI 192.168.201.8@tcp [ 2086.778284] LNet: Removed LNI 192.168.201.8@tcp2 [ 2086.859807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.279971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.309471] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2087.981540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2088.006597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2088.954205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.976809] LNetError: 118906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2088.980749] LNetError: 118906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2088.984945] LNet: Removed LNI 10.1.2.3@tcp [ 2088.995983] LNet: Removed LNI 192.168.201.8@tcp [ 2089.012349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.090387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2089.137919] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2089.155018] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2089.450269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2089.471956] LNet: Removed LNI 192.168.201.8@tcp [ 2089.484669] LNet: Removed LNI 10.1.2.3@tcp [ 2096.003935] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:20:10 (1786018810) [ 2096.221335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.294485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.416657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.463200] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2097.279372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2097.334943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.380603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.416767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.448900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.473985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.506346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.541374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.576584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.611705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.638712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.8@tcp [ 2097.769557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2104.023815] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:20:19 (1786018819) [ 2104.159606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.257868] LNet: Removed LNI 192.168.201.8@tcp [ 2105.284312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.416485] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2105.525078] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2105.532951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2105.538755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2107.950389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2107.992621] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2108.719754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2108.762240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2110.267783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.8@tcp [ 2110.311317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2110.387309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.8@tcp -m PUT -e local_timeout -r 1 [ 2110.451874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2110.509444] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2110.513200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2110.719499] LNetError: 120312:0:(lib-move.c:4801:lnet_parse()) 192.168.201.8@tcp, src 192.168.201.8@tcp: Dropping PUT (error -108 looking up sender) [ 2111.776598] LNet: Removed LNI 10.1.2.3@tcp [ 2112.808803] LNet: Removed LNI 192.168.201.8@tcp [ 2113.247496] Key type .llcrypt unregistered [ 2113.249705] Key type ._llcrypt unregistered [ 2124.923708] Key type ._llcrypt registered [ 2124.926436] Key type .llcrypt registered [ 2126.058860] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:20:40 (1786018840) [ 2126.472381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.491620] alg: No test for adler32 (adler32-zlib) [ 2127.370809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.813856] LNet: 121629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.286484] Key type .llcrypt unregistered [ 2128.288459] Key type ._llcrypt unregistered [ 2134.344432] Key type ._llcrypt registered [ 2134.348376] Key type .llcrypt registered [ 2134.548366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2134.569494] alg: No test for adler32 (adler32-zlib) [ 2135.445346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2135.465911] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2143.924906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 2144.227576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2145.058136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.108@tcp [ 2145.467526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2145.520954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2145.556141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2145.640168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2145.694867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2145.700176] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.108@tcp [ 2145.756692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2145.806740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2145.892841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2145.928473] LNet: 122080:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2145.972019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2145.998419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2146.034493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2146.064318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2146.112645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2146.141606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2146.771382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2146.946169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2146.966734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2146.989418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2147.017994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2147.049807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2147.076487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2147.107973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2147.127191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2147.140283] LNet: 122197:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2147.156542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2147.182518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2147.213808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2147.236156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2147.267343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2147.300056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2147.880488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.108@tcp [ 2148.169347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2148.195124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2148.225201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2148.254744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2148.280671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2148.311231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2148.335652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2148.353553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2148.366427] LNet: 122314:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2148.379816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2148.408101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2148.425805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2148.449818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2148.481626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2148.503349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2149.095681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2149.273110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2149.290855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2149.312306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2149.337922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2149.364936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2149.399956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2149.460714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2149.515663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2149.557300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2149.579734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2149.598548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2149.619782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2149.642684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2149.684971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2149.957936] LNet: 122477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.964570] LNetError: 122477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.981152] LNet: Removed LNI 192.168.201.8@tcp [ 2150.504512] Key type .llcrypt unregistered [ 2150.510765] Key type ._llcrypt unregistered [ 2171.223320] Key type ._llcrypt registered [ 2171.226080] Key type .llcrypt registered [ 2172.986730] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:21:27 (1786018887) [ 2173.651196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.682265] alg: No test for adler32 (adler32-zlib) [ 2174.685460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.510780] LNet: 123366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.862664] Key type .llcrypt unregistered [ 2176.871432] Key type ._llcrypt unregistered [ 2186.894167] Key type ._llcrypt registered [ 2186.911393] Key type .llcrypt registered [ 2187.158779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.180066] alg: No test for adler32 (adler32-zlib) [ 2188.114188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2188.144409] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2196.927760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.108@tcp [ 2206.455814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.510619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2206.554663] LNet: Added LNI 192.168.201.8@tcp1 [8/256/0/180] [ 2207.721592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2208.280619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.108@tcp [ 2208.619267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2208.672441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2208.674636] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.108@tcp [ 2208.739483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2208.804208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2208.840661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2208.903854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2208.952121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2208.995675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2209.039401] LNet: 124020:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.084101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2209.133750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2209.188437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2209.251539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2209.302375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2209.387165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2210.229582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2210.621350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2210.655617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2210.695605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2210.757622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2210.824337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2210.862761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2210.898858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2210.944615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2210.969749] LNet: 124137:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.007497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2211.065948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2211.116613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2211.160652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2211.212584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2211.256941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2212.056104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.108@tcp [ 2212.280938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2212.318618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2212.361143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2212.395954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2212.432652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2212.484812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2212.528168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2212.568460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2212.581470] LNet: 124254:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.594462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2212.631498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2212.659980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2212.695275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2212.714100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2212.748410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2213.339988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2213.586175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2213.610922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2213.633975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2213.670907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2213.692198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2213.716755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2213.746716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2213.821768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2213.865917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2213.921619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2213.969784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2214.010437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2214.054397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2214.106425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2219.330607] LNet: 124452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.336808] LNetError: 124452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.383576] LNet: Removed LNI 192.168.201.8@tcp [ 2220.386725] LNetError: 124452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.399583] LNet: Removed LNI 192.168.201.8@tcp1 [ 2220.829814] Key type .llcrypt unregistered [ 2220.831795] Key type ._llcrypt unregistered [ 2239.388484] Key type ._llcrypt registered [ 2239.392346] Key type .llcrypt registered [ 2240.558282] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:22:35 (1786018955) [ 2241.842981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2243.164703] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:22:38 (1786018958) [ 2250.361588] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:22:45 (1786018965) [ 2258.193382] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:22:52 (1786018972) [ 2264.768665] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:23:00 (1786018980) [ 2270.624643] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:23:05 (1786018985) [ 2277.406671] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:23:12 (1786018992) [ 2284.041181] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:23:18 (1786018998) [ 2290.696779] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:23:25 (1786019005) [ 2297.824725] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:23:32 (1786019012) [ 2306.391974] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:23:40 (1786019020) [ 2312.687939] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:23:47 (1786019027) [ 2320.236239] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:23:55 (1786019035) [ 2327.571076] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:24:01 (1786019041) [ 2333.835742] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:24:08 (1786019048) [ 2341.326582] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:24:16 (1786019056) [ 2348.130814] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:24:22 (1786019062) [ 2356.014964] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:24:30 (1786019070) [ 2363.734436] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:24:38 (1786019078) [ 2364.134748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2364.162535] alg: No test for adler32 (adler32-zlib) [ 2365.134077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.326498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.360500] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2366.349725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.381214] LNet: 135452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.394380] LNetError: 135452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2366.416034] LNet: Removed LNI 192.168.201.8@tcp [ 2366.449859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.513483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2366.532832] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2372.495171] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:24:47 (1786019087) [ 2378.849992] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:24:53 (1786019093) [ 2385.948584] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:25:01 (1786019101) [ 2392.478523] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:25:07 (1786019107) [ 2398.638962] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:25:13 (1786019113) [ 2406.099651] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:25:20 (1786019120) [ 2413.814454] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:25:27 (1786019127) [ 2423.365353] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:25:37 (1786019137) [ 2430.771864] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:25:45 (1786019145) [ 2438.669673] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:25:53 (1786019153) [ 2446.414578] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:26:01 (1786019161) [ 2453.718213] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:26:08 (1786019168) [ 2461.376832] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2205 sec ========== 08:26:15 (1786019175) [ 2461.678206] LNet: 142919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.687398] LNetError: 142919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.703763] LNet: Removed LNI 192.168.201.8@tcp [ 2462.230711] Key type .llcrypt unregistered [ 2462.232736] Key type ._llcrypt unregistered [ 2477.375183] Key type ._llcrypt registered [ 2477.378671] Key type .llcrypt registered [ 2477.855334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2477.870751] alg: No test for adler32 (adler32-zlib) [ 2479.062657] Lustre: Lustre: Build Version: 2.17.56_2_ge0d4711 [ 2479.322929] LNet: Added LNI 192.168.201.8@tcp [8/256/0/180] [ 2481.095718] Key type lgssc registered [ 2482.768533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2548.598310] Lustre: Mounted lustre-client [ 2553.443353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2561.052692] Lustre: Unmounted lustre-client [ 2581.236572] Key type lgssc unregistered [ 2581.571717] LNet: 145847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2581.578942] LNetError: 145847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.644110] LNet: Removed LNI 192.168.201.8@tcp [ 2583.348134] Key type .llcrypt unregistered [ 2583.352245] Key type ._llcrypt unregistered