[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 451441084 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001084] APIC: Switch to symmetric I/O mode setup [ 0.003447] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008612] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012046] LSM: Security Framework initializing [ 0.014051] Yama: becoming mindful. [ 0.015046] SELinux: Initializing. [ 0.016161] *** VALIDATE selinux *** [ 0.028394] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.036000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036226] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038000] *** VALIDATE tmpfs *** [ 0.041160] *** VALIDATE proc *** [ 0.042196] *** VALIDATE cgroup *** [ 0.043015] *** VALIDATE cgroup2 *** [ 0.045040] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.047146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.048010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.049032] Spectre V2 : User space: Vulnerable [ 0.050014] Speculative Store Bypass: Vulnerable [ 0.056402] debug: unmapping init [mem 0xffffffff9c259000-0xffffffff9c260fff] [ 0.059641] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.061335] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.062580] ... version: 2 [ 0.063018] ... bit width: 48 [ 0.064027] ... generic registers: 4 [ 0.065016] ... value mask: 0000ffffffffffff [ 0.066019] ... max period: 00007fffffffffff [ 0.067616] ... fixed-purpose events: 3 [ 0.068019] ... event mask: 000000070000000f [ 0.070331] rcu: Hierarchical SRCU implementation. [ 0.075650] smp: Bringing up secondary CPUs ... [ 0.077876] x86: Booting SMP configuration: [ 0.078054] .... node #0, CPUs: #1 #2 #3 [ 0.105030] smp: Brought up 1 node, 4 CPUs [ 0.107019] smpboot: Max logical packages: 1 [ 0.108016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.173021] node 0 deferred pages initialised in 58ms [ 0.183249] devtmpfs: initialized [ 0.186250] x86/mm: Memory block size: 128MB [ 0.191481] gcov: version magic: 0x41383552 [ 0.200215] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.208111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.217263] pinctrl core: initialized pinctrl subsystem [ 0.223177] [ 0.228000] ************************************************************* [ 0.236015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241012] ** ** [ 0.245013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249017] ** ** [ 0.252022] ** This means that this kernel is built to expose internal ** [ 0.259014] ** IOMMU data structures, which may compromise security on ** [ 0.267013] ** your system. ** [ 0.276014] ** ** [ 0.283014] ** If you see this message and you are not debugging the ** [ 0.287016] ** kernel, report this immediately to your vendor! ** [ 0.292014] ** ** [ 0.300016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.303014] ************************************************************* [ 0.321194] NET: Registered protocol family 16 [ 0.328012] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.335077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.343534] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.349105] cpuidle: using governor menu [ 0.354746] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.366151] PCI: Using configuration type 1 for base access [ 0.371834] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.386046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.387024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.397073] cryptd: max_cpu_qlen set to 1000 [ 0.401390] ACPI: Added _OSI(Module Device) [ 0.403052] ACPI: Added _OSI(Processor Device) [ 0.406015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.407015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.413699] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.441665] ACPI: Interpreter enabled [ 0.443139] ACPI: PM: (supports S0 S3 S4 S5) [ 0.446014] ACPI: Using IOAPIC for interrupt routing [ 0.452039] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.458611] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.486000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.491045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.498024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.505204] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.515442] acpiphp: Slot [2] registered [ 0.519792] acpiphp: Slot [5] registered [ 0.522790] acpiphp: Slot [6] registered [ 0.525528] acpiphp: Slot [3] registered [ 0.530141] acpiphp: Slot [4] registered [ 0.535174] acpiphp: Slot [7] registered [ 0.536000] acpiphp: Slot [8] registered [ 0.537270] acpiphp: Slot [9] registered [ 0.539116] acpiphp: Slot [10] registered [ 0.545929] acpiphp: Slot [11] registered [ 0.549123] acpiphp: Slot [12] registered [ 0.552132] acpiphp: Slot [13] registered [ 0.555098] acpiphp: Slot [14] registered [ 0.557129] acpiphp: Slot [15] registered [ 0.559125] acpiphp: Slot [16] registered [ 0.567214] acpiphp: Slot [17] registered [ 0.569159] acpiphp: Slot [18] registered [ 0.573139] acpiphp: Slot [19] registered [ 0.580298] acpiphp: Slot [20] registered [ 0.583962] acpiphp: Slot [21] registered [ 0.586133] acpiphp: Slot [22] registered [ 0.589118] acpiphp: Slot [23] registered [ 0.593131] acpiphp: Slot [24] registered [ 0.596161] acpiphp: Slot [25] registered [ 0.598418] acpiphp: Slot [26] registered [ 0.604338] acpiphp: Slot [27] registered [ 0.607117] acpiphp: Slot [28] registered [ 0.609201] acpiphp: Slot [29] registered [ 0.612643] acpiphp: Slot [30] registered [ 0.617065] acpiphp: Slot [31] registered [ 0.620285] PCI host bridge to bus 0000:00 [ 0.622027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.627023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.632039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.638024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.644023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.647039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.649249] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.654490] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.659830] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.671014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.681867] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.687080] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.691020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.695077] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.698910] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.703000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.713053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.717169] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 13671 usecs [ 0.720000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.734017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.751016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.758014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.765995] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.777016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.792000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.825018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.839000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.853000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.873016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.898000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.918987] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.925167] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.929821] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.935494] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.938257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.954237] iommu: Default domain type: Passthrough [ 0.961681] SCSI subsystem initialized [ 0.964305] ACPI: bus type USB registered [ 0.966161] usbcore: registered new interface driver usbfs [ 0.969082] usbcore: registered new interface driver hub [ 0.972088] usbcore: registered new device driver usb [ 0.976165] pps_core: LinuxPPS API ver. 1 registered [ 0.977013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.984118] PTP clock support registered [ 0.988093] EDAC MC: Ver: 3.0.0 [ 0.989092] PCI: Using ACPI for IRQ routing [ 0.990783] NetLabel: Initializing [ 0.992015] NetLabel: domain hash size = 128 [ 0.996263] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.000079] NetLabel: unlabeled traffic allowed by default [ 1.003533] vgaarb: loaded [ 1.012274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.015016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.040104] clocksource: Switched to clocksource kvm-clock [ 1.230366] VFS: Disk quotas dquot_6.6.0 [ 1.232388] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.243802] *** VALIDATE ramfs *** [ 1.246619] *** VALIDATE hugetlbfs *** [ 1.251920] pnp: PnP ACPI init [ 1.258573] pnp: PnP ACPI: found 6 devices [ 1.334725] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.350122] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.354254] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.359466] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.364847] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.368828] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.374482] NET: Registered protocol family 2 [ 1.379163] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.389479] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.406900] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.423640] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.439690] TCP: Hash tables configured (established 65536 bind 65536) [ 1.454362] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.457494] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.469333] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.475699] NET: Registered protocol family 1 [ 1.484447] RPC: Registered named UNIX socket transport module. [ 1.486318] RPC: Registered udp transport module. [ 1.495538] RPC: Registered tcp transport module. [ 1.503309] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.514182] NET: Registered protocol family 44 [ 1.519660] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.524368] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.530739] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.539294] PCI: CLS 0 bytes, default 64 [ 1.552325] Unpacking initramfs... [ 4.203611] debug: unmapping init [mem 0xffff92973cc64000-0xffff92973ffcffff] [ 4.208385] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.211305] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.215161] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.457280] Initialise system trusted keyrings [ 5.459553] Key type blacklist registered [ 5.465878] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.494858] zbud: loaded [ 5.511844] *** VALIDATE nfs *** [ 5.515727] *** VALIDATE nfs4 *** [ 5.520537] pstore: using deflate compression [ 5.540478] Platform Keyring initialized [ 5.874143] NET: Registered protocol family 38 [ 5.877776] Key type asymmetric registered [ 5.880901] Asymmetric key parser 'x509' registered [ 5.883413] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.889247] io scheduler mq-deadline registered [ 5.890971] io scheduler kyber registered [ 5.895878] io scheduler bfq registered [ 5.920476] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.930556] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.935284] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.946866] ACPI: Power Button [PWRF] [ 5.964710] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.992877] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.017775] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.049354] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.100434] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.118788] Non-volatile memory driver v1.3 [ 6.127054] Linux agpgart interface v0.103 [ 6.193384] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 6.201939] vda: detected capacity change from 0 to 75079680 [ 6.241656] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.246050] vdb: detected capacity change from 0 to 1073741824 [ 6.266879] libphy: Fixed MDIO Bus: probed [ 6.305266] usbcore: registered new interface driver usbserial_generic [ 6.309977] usbserial: USB Serial support registered for generic [ 6.320772] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.332472] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.340381] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.345345] mousedev: PS/2 mouse device common for all mice [ 6.360368] rtc_cmos 00:05: RTC can wake from S4 [ 6.371693] rtc_cmos 00:05: registered as rtc0 [ 6.378354] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.382575] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.389272] intel_pstate: CPU model not supported [ 6.400480] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.411255] hid: raw HID events driver (C) Jiri Kosina [ 6.420650] usbcore: registered new interface driver usbhid [ 6.424771] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.426921] usbhid: USB HID core driver [ 6.427432] drop_monitor: Initializing network drop monitor service [ 6.444815] Initializing XFRM netlink socket [ 6.448636] NET: Registered protocol family 10 [ 6.467708] Segment Routing with IPv6 [ 6.471414] NET: Registered protocol family 17 [ 6.478488] mpls_gso: MPLS GSO support [ 6.501450] RAS: Correctable Errors collector initialized. [ 6.506319] AVX version of gcm_enc/dec engaged. [ 6.510027] AES CTR mode by8 optimization enabled [ 6.687382] sched_clock: Marking stable (6687362132, 0)->(8151134090, -1463771958) [ 6.698150] registered taskstats version 1 [ 6.705220] Loading compiled-in X.509 certificates [ 6.710336] zswap: loaded using pool lzo/zbud [ 6.755661] Key type big_key registered [ 6.788349] Key type encrypted registered [ 6.790627] ima: No TPM chip found, activating TPM-bypass! [ 6.794404] ima: Allocated hash algorithm: sha1 [ 6.797348] ima: No architecture policies found [ 6.799672] evm: Initialising EVM extended attributes: [ 6.802952] evm: security.selinux [ 6.804410] evm: security.ima [ 6.805785] evm: security.capability [ 6.807662] evm: HMAC attrs: 0x1 [ 6.810970] rtc_cmos 00:05: setting system clock to 2026-09-02 06:01:51 UTC (1788328911) [ 6.820494] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 6.828685] debug: unmapping init [mem 0xffffffff9bf82000-0xffffffff9c258fff] [ 6.837252] Write protecting the kernel read-only data: 28672k [ 6.858833] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 6.865641] debug: unmapping init [mem 0xffffffff9af14000-0xffffffff9affffff] [ 6.999280] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.023782] systemd[1]: Detected virtualization kvm. [ 7.025627] systemd[1]: Detected architecture x86-64. [ 7.028613] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.100824] systemd[1]: No hostname configured. [ 7.103342] systemd[1]: Set hostname to . [ 7.106243] random: systemd: uninitialized urandom read (16 bytes read) [ 7.110395] systemd[1]: Initializing machine ID from random generator. [ 7.213970] random: ln: uninitialized urandom read (6 bytes read) [ 7.404620] random: systemd: uninitialized urandom read (16 bytes read) [ 7.408345] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.417814] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.434648] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.151774] device-mapper: uevent: version 1.0.3 [ 9.159208] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.922052] virtio_net virtio0 ens2: renamed from eth0 [ 11.699879] scsi host0: ata_piix [ 11.775613] scsi host1: ata_piix [ 11.775822] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.775830] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 16.325879] random: crng init done [ 16.329359] random: 7 urandom warning(s) missed due to ratelimiting [ 17.320375] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.835755] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.753420] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.753943] SELinux: Disabled at runtime. [ 22.848178] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.865496] systemd[1]: Detected virtualization kvm. [ 22.867624] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.772471] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.783267] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.799196] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.816815] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.830180] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.883577] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.891688] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 25.239742] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.664526] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.511523] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.660601] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.330124] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.503846] EDAC sbridge: Ver: 1.1.2 [ 32.266272] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit)[ 33.169361] NFS: Registering the id_resolver key type [ 33.173163] Key type id_resolver registered [ 33.173176] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-client login: [ 100.772805] libcfs: loading out-of-tree module taints kernel. [ 101.215134] Key type ._llcrypt registered [ 101.233513] Key type .llcrypt registered [ 101.776311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 101.784682] alg: No test for adler32 (adler32-zlib) [ 103.653830] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 104.931065] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 106.824475] Key type lgssc registered [ 109.610709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 171.982258] hrtimer: interrupt took 5163770 ns [ 245.237735] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c [ 250.597854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.257081] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing check_logdir /tmp/testlogs/ [ 270.240566] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing yml_node [ 270.815141] Lustre: lustre-OST0000-osc-ffff929785593000: disconnect after 23s idle [ 275.342060] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 278.631699] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 281.871315] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 283.583608] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 2 02:06:26 EDT 2026 [ 302.626858] Lustre: Unmounted lustre-client [ 332.415472] Key type lgssc unregistered [ 332.705381] LNet: 8429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.712772] LNetError: 8429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.736991] LNet: Removed LNI 192.168.203.59@tcp [ 333.788330] Key type .llcrypt unregistered [ 333.790245] Key type ._llcrypt unregistered [ 361.603909] Key type ._llcrypt registered [ 361.612952] Key type .llcrypt registered [ 361.858739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 361.875594] alg: No test for adler32 (adler32-zlib) [ 362.823473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 362.918940] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 362.947566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 367.041025] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 367.130349] LNet: 9302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.138766] LNetError: 9302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 367.155221] LNet: Removed LNI 192.168.203.59@tcp [ 367.756723] Key type .llcrypt unregistered [ 367.761518] Key type ._llcrypt unregistered [ 374.380983] Key type ._llcrypt registered [ 374.383192] Key type .llcrypt registered [ 376.074305] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:07:58 (1788329278) [ 376.346047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 376.370807] alg: No test for adler32 (adler32-zlib) [ 377.327707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.431480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.463111] LNet: 9717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.478426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.529989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 377.545808] LNet: 9724:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 385.343809] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:08:08 (1788329288) [ 385.457665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.505102] LNet: 10297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.527661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.619845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 392.128663] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:08:15 (1788329295) [ 392.316347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.342781] LNet: 10889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.367285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.427351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 400.333216] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:08:23 (1788329303) [ 400.715417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.735970] LNet: 11473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.762549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.913150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 409.037965] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:08:32 (1788329312) [ 409.132978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.156137] LNet: 12052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.197336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.325809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 409.670452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 415.321386] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:08:38 (1788329318) [ 415.511929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.576813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.701445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 416.232554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 416.702817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 422.572401] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:08:46 (1788329326) [ 422.752883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.787783] LNet: 13245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.794375] LNet: 13245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 422.815708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.900538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 429.077556] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:08:52 (1788329332) [ 429.160339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.190202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.258390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 429.288927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 429.724751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 429.751444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 430.045465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 430.071358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 430.396800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 430.427899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 430.671329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 430.694658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.022059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 431.053086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.362802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 431.386949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.766754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 431.792686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 432.185254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 432.211517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 432.552586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 432.587622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 433.010273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 433.049690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 433.591878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 433.629356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 434.009106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 434.076207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 440.122334] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:09:03 (1788329343) [ 440.334884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.418186] LNet: 14509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.431697] LNet: 14509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 440.471557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.556839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 440.590825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 447.464190] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:09:10 (1788329350) [ 447.676209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.764482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.884470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 447.952764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 455.642080] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:09:18 (1788329358) [ 455.787622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.873597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.999697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 456.039609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 466.061472] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:09:28 (1788329368) [ 466.231500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.318037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.471081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 466.537858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 476.080612] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:09:38 (1788329378) [ 476.607168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.675224] LNet: 16857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.680129] LNet: 16857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 476.722124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.828274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 476.886658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 485.048833] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:09:48 (1788329388) [ 485.133414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.175172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.291674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.365978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 492.371460] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:09:55 (1788329395) [ 492.515532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.564265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.658481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.188450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.782029] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:10:02 (1788329402) [ 499.904753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.975981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.234932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.526707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 506.949543] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:10:10 (1788329410) [ 507.151824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.181858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.299364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.575222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.157752] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:10:18 (1788329418) [ 515.510313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.557216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.021772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.357349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.761362] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:10:26 (1788329426) [ 523.866716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.920968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.036239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.116425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.398696] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:10:34 (1788329434) [ 532.585497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.632392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.716450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.753782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 541.610705] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:10:44 (1788329444) [ 541.737557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.763757] LNet: 21709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.770533] LNet: 21709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 541.795648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.921957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 548.612531] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:10:51 (1788329451) [ 548.861170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.927477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.033544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 556.680970] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:10:59 (1788329459) [ 556.817324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.875642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.986644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 557.535968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 565.204917] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:11:08 (1788329468) [ 565.384725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.429238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.549895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 565.582828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 572.933538] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:11:15 (1788329475) [ 573.035888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.090572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.220718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 573.273657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 579.997751] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:11:23 (1788329483) [ 580.221427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.260452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.354438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 580.403856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 587.071875] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:11:30 (1788329490) [ 587.155415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.209198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.313052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 587.353596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 595.409758] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:11:38 (1788329498) [ 595.651195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.745332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.859666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 595.991863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 596.031497] LNetError: 25833:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 596.110854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 596.184815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 603.716995] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:11:46 (1788329506) [ 603.874219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.933724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.030137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KR0aNukYIJ [ 604.171501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KR0aNukYIJ [ 604.237353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KR0aNukYIJ [ 610.260607] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:11:53 (1788329513) [ 610.446376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.479597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.565210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 610.601489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 610.658479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 617.032385] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:12:00 (1788329520) [ 617.433906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.471602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.533651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 617.557252] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 619.309366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 620.629263] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 622.626616] Key type lgssc registered [ 623.805570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 627.214589] LNetError: 27595:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.59@tcp99 has no available nets [ 629.767653] Key type lgssc unregistered [ 630.121877] LNetError: 28701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.139981] LNet: Removed LNI 192.168.203.59@tcp [ 630.826759] Key type .llcrypt unregistered [ 630.834754] Key type ._llcrypt unregistered [ 637.000709] Key type ._llcrypt registered [ 637.005666] Key type .llcrypt registered [ 638.296121] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:12:21 (1788329541) [ 638.871448] Key type .llcrypt unregistered [ 638.875364] Key type ._llcrypt unregistered [ 645.479970] Key type ._llcrypt registered [ 645.483920] Key type .llcrypt registered [ 645.678212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.693485] alg: No test for adler32 (adler32-zlib) [ 646.632452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.738604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 646.772461] LNet: 29610:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 646.778885] LNet: 29610:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 646.783990] LNet: 29610:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 646.792513] LNet: Message forwarding will be enabled in 2s [ 649.887471] LNet: Message forwarding enabled [ 650.322446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 650.338315] LNet: Message forwarding will stop in 2s to 6s [ 653.985465] LNet: Message forwarding disabled [ 654.918468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 654.959619] LNet: 29708:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 654.978096] LNet: 29708:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 655.004526] LNet: 29708:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 655.026453] LNet: Message forwarding will be enabled in 2s [ 655.204854] LNet: 29721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.919963] Key type .llcrypt unregistered [ 655.924768] Key type ._llcrypt unregistered [ 667.524636] Key type ._llcrypt registered [ 667.527037] Key type .llcrypt registered [ 669.126873] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:12:52 (1788329572) [ 669.411816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.426515] alg: No test for adler32 (adler32-zlib) [ 670.308349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.412853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 676.360513] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:12:59 (1788329579) [ 676.530353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.570355] LNet: 31119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.624561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.698450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 676.735976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 676.768533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 676.807795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 676.856287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 676.918914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 676.949320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 676.975629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 677.011915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 677.049558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 677.095550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 677.138711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 677.186407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 677.254232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 677.328338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 677.369461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 677.420983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 677.497149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 677.548920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 677.587829] LNetError: 31165:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 683.763313] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:13:07 (1788329587) [ 683.915877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.955935] LNet: 31732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.991126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.098232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.335439] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:13:13 (1788329593) [ 691.802362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg359-server [ 693.363901] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:13:16 (1788329596) [ 693.513588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.538254] LNet: 32667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.578103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.629543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.647666] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 694.276100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[60-64]@tcp [ 694.313503] LNet: 32801:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 694.371572] LNetError: 32672:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 694.396278] LNetError: 32672:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.60@tcp has gone from up to down [ 697.443662] LNetError: 32678:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.64@tcp has gone from up to down [ 700.785661] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:13:23 (1788329603) [ 702.718295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg359-server [ 704.699276] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:13:27 (1788329607) [ 704.812622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.861870] LNet: 33730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.877778] LNetError: 33730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.922856] LNet: Removed LNI 192.168.203.59@tcp [ 704.989545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.053951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.085575] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 705.768200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[60-65/2]@tcp [ 705.798188] LNet: 33853:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 705.806242] LNet: 33853:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 705.846713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[60-65/2]@tcp [ 713.043774] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:13:36 (1788329616) [ 713.288341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.338806] LNet: 34446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.344346] LNetError: 34446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.358731] LNet: Removed LNI 192.168.203.59@tcp [ 713.405043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.513510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 713.569238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.615763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.675575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.715676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.741894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.807857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 713.942486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.000238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.093545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.194834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.248556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.332808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 714.473964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.551917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.623119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.739649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.790738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.861838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 714.991200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.018961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.076729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.148662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.190725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.239253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 715.337904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.415519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.506924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.577630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.634500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.722693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 715.785533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.852122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.909785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.738198] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:13:46 (1788329626) [ 725.778665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg359-server [ 727.817213] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:13:51 (1788329631) [ 730.056330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg359-server [ 732.142257] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:13:55 (1788329635) [ 732.295941] LNet: 35844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.309548] LNet: 35844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 733.045404] Key type .llcrypt unregistered [ 733.048613] Key type ._llcrypt unregistered [ 739.045748] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.261774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.281829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.307363] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.050138] Key type ._llcrypt registered [ 742.060570] Key type .llcrypt registered [ 742.317888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.337990] alg: No test for adler32 (adler32-zlib) [ 743.255569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.345296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.380105] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 743.990620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 744.027489] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 745.826775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 745.857812] LNet: Removed LNI 10.1.2.3@tcp [ 745.990897] LNet: 36566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.001986] LNetError: 36566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.016994] LNet: Removed LNI 192.168.203.59@tcp [ 746.716656] Key type .llcrypt unregistered [ 746.721815] Key type ._llcrypt unregistered [ 758.873603] Key type ._llcrypt registered [ 758.877139] Key type .llcrypt registered [ 760.075309] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:14:23 (1788329663) [ 760.570267] Key type .llcrypt unregistered [ 760.572146] Key type ._llcrypt unregistered [ 766.176189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.314410] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.333599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.348271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.868191] Key type ._llcrypt registered [ 767.869971] Key type .llcrypt registered [ 768.135405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.169564] alg: No test for adler32 (adler32-zlib) [ 769.075255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.145789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.171292] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 769.782162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 769.798311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 772.203793] LNet: 38047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.216980] LNetError: 38047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 772.224807] LNet: Removed LNI 10.1.2.3@tcp [ 772.234805] LNet: Removed LNI 192.168.203.59@tcp [ 772.893433] Key type .llcrypt unregistered [ 772.895611] Key type ._llcrypt unregistered [ 784.156767] Key type ._llcrypt registered [ 784.159462] Key type .llcrypt registered [ 785.443483] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:14:48 (1788329688) [ 786.008586] Key type .llcrypt unregistered [ 786.010727] Key type ._llcrypt unregistered [ 792.126839] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.272303] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.289605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.297861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.854586] Key type ._llcrypt registered [ 794.864942] Key type .llcrypt registered [ 795.139865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.158996] alg: No test for adler32 (adler32-zlib) [ 796.130910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.202381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 796.227942] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 797.995831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 798.020033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.302875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 800.337062] LNet: Removed LNI 10.1.2.3@tcp [ 800.356230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 800.388044] LNet: Removed LNI 10.1.2.31@tcp [ 800.436567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 800.471640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.404310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 801.435608] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 802.546793] LNet: 39959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.557048] LNetError: 39959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 802.568492] LNet: Removed LNI 10.1.2.31@tcp [ 802.601165] LNet: Removed LNI 10.1.2.3@tcp [ 803.289634] Key type .llcrypt unregistered [ 803.293697] Key type ._llcrypt unregistered [ 815.107958] Key type ._llcrypt registered [ 815.110966] Key type .llcrypt registered [ 816.072666] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:15:19 (1788329719) [ 816.420954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.434743] alg: No test for adler32 (adler32-zlib) [ 817.336501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.388114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.411061] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 818.068951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.086229] LNet: 40909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.096735] LNetError: 40909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.109951] LNet: Removed LNI 192.168.203.59@tcp [ 818.127352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.191191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.59@tcp [ 818.218932] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 818.775589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.59@tcp [ 818.812138] LNet: Removed LNI 192.168.203.59@tcp [ 818.886563] LNet: 41044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.412403] Key type .llcrypt unregistered [ 819.414178] Key type ._llcrypt unregistered [ 828.548069] Key type ._llcrypt registered [ 828.554299] Key type .llcrypt registered [ 829.528417] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:15:33 (1788329733) [ 831.025971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg359-server [ 832.699704] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:15:36 (1788329736) [ 833.199439] Key type .llcrypt unregistered [ 833.204248] Key type ._llcrypt unregistered [ 838.469808] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 838.546552] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 838.557759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 838.567612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 841.119262] Key type ._llcrypt registered [ 841.121295] Key type .llcrypt registered [ 841.307203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.328168] alg: No test for adler32 (adler32-zlib) [ 842.208534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.267764] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 842.978818] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 844.804565] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.809767] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.815183] LNet: Removed LNI 192.168.203.59@tcp1 [ 844.829591] LNet: Removed LNI 10.1.2.3@tcp2 [ 845.463489] Key type .llcrypt unregistered [ 845.465643] Key type ._llcrypt unregistered [ 856.537349] Key type ._llcrypt registered [ 856.540355] Key type .llcrypt registered [ 858.315971] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:16:01 (1788329761) [ 858.739571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.755654] alg: No test for adler32 (adler32-zlib) [ 859.674930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.736801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.782814] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 860.601193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.645620] LNet: 43857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.655768] LNetError: 43857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.668798] LNet: Removed LNI 192.168.203.59@tcp [ 860.690060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.742916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 860.762100] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 868.346495] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:16:11 (1788329771) [ 868.420137] LNet: 44440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.432908] LNetError: 44440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.452810] LNet: Removed LNI 192.168.203.59@tcp [ 869.059075] Key type .llcrypt unregistered [ 869.068287] Key type ._llcrypt unregistered [ 875.261744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 875.402518] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 875.414154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.428122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.184805] Key type ._llcrypt registered [ 878.189150] Key type .llcrypt registered [ 878.425981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.448871] alg: No test for adler32 (adler32-zlib) [ 879.412767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.470746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.508817] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 880.364354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 880.401467] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.703541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.723202] LNet: 45188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.731707] LNetError: 45188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.738714] LNet: Removed LNI 10.1.2.3@tcp [ 882.750947] LNet: Removed LNI 192.168.203.59@tcp [ 882.793155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.859652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 882.895341] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 882.899902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.387936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.412240] LNet: 45215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.419038] LNetError: 45215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.428969] LNetError: 45215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.441026] LNet: Removed LNI 10.1.2.3@tcp [ 883.461221] LNet: Removed LNI 192.168.203.59@tcp [ 883.483244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.549747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 883.587959] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 883.598839] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.133718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 884.154817] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 885.155801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.193484] LNet: 45377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.200392] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.210161] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.216261] LNet: Removed LNI 10.1.2.3@tcp [ 885.218231] LNet: Removed LNI 192.168.203.59@tcp [ 885.246993] LNet: Removed LNI 192.168.203.59@tcp2 [ 885.278988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.329596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 885.359052] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 885.364654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.370463] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 892.202899] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:16:35 (1788329795) [ 892.312658] LNet: 45969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.332391] LNetError: 45969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.341456] LNetError: 45969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 892.353451] LNet: Removed LNI 10.1.2.3@tcp [ 892.355546] LNet: Removed LNI 192.168.203.59@tcp [ 892.366121] LNet: Removed LNI 192.168.203.59@tcp2 [ 893.048085] Key type .llcrypt unregistered [ 893.050908] Key type ._llcrypt unregistered [ 899.085434] Key type ._llcrypt registered [ 899.092733] Key type .llcrypt registered [ 899.351662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.362930] alg: No test for adler32 (adler32-zlib) [ 900.328140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.406855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.442311] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 901.259917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.299491] LNet: 46366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.311548] LNetError: 46366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.341131] LNet: Removed LNI 192.168.203.59@tcp [ 901.384864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.461579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 901.488344] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 902.178229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.205044] LNet: 46396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.218623] LNetError: 46396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.235342] LNet: Removed LNI 192.168.203.59@tcp [ 902.261562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.335872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 902.382558] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 902.871648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.909847] LNet: Removed LNI 192.168.203.59@tcp [ 902.952598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.083207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 903.123705] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 903.798477] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 904.009687] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 904.013195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 904.020795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 906.445465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.479383] LNet: 46507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.490534] LNet: 46507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 906.505665] LNetError: 46507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.519168] LNetError: 46507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 906.546729] LNet: Removed LNI 192.168.203.59@tcp [ 906.564944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.624578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 906.648594] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 912.914331] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:16:55 (1788329815) [ 913.079152] LNet: 47101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.093862] LNetError: 47101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.138573] LNet: Removed LNI 192.168.203.59@tcp [ 913.872852] Key type .llcrypt unregistered [ 913.876683] Key type ._llcrypt unregistered [ 920.709545] Key type ._llcrypt registered [ 920.712503] Key type .llcrypt registered [ 920.898550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.916708] alg: No test for adler32 (adler32-zlib) [ 921.802407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.850337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.873776] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 922.707135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.733446] LNet: 47501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.744616] LNetError: 47501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.763647] LNet: Removed LNI 192.168.203.59@tcp [ 922.820516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.885490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 922.918614] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 929.089728] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:17:12 (1788329832) [ 929.230097] LNet: 48088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.236106] LNetError: 48088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.279306] LNet: Removed LNI 192.168.203.59@tcp [ 930.016510] Key type .llcrypt unregistered [ 930.019974] Key type ._llcrypt unregistered [ 935.854174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 935.971862] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 935.976295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 935.985937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 938.557696] Key type ._llcrypt registered [ 938.561100] Key type .llcrypt registered [ 938.725298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.738251] alg: No test for adler32 (adler32-zlib) [ 939.711205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.771919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.806604] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 940.834677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.866935] LNet: 48540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.880309] LNetError: 48540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.906360] LNet: Removed LNI 192.168.203.59@tcp [ 940.929667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.995864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 941.044370] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 948.323790] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:17:31 (1788329851) [ 948.404450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.441290] LNet: 49131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.447381] LNetError: 49131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.468108] LNet: Removed LNI 192.168.203.59@tcp [ 948.498565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.643383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 954.226723] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:17:37 (1788329857) [ 954.394996] LNet: 49720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.974692] Key type .llcrypt unregistered [ 954.976600] Key type ._llcrypt unregistered [ 960.774601] Key type ._llcrypt registered [ 960.783116] Key type .llcrypt registered [ 961.080910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.097128] alg: No test for adler32 (adler32-zlib) [ 962.042656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.092424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.122382] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 963.017770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.037804] LNet: 50116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.043939] LNetError: 50116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.060125] LNet: Removed LNI 192.168.203.59@tcp [ 963.075765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.143512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 963.189065] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 970.661149] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:17:53 (1788329873) [ 970.776772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.843251] LNet: 50703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.856716] LNetError: 50703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.894160] LNet: Removed LNI 192.168.203.59@tcp [ 970.939049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.026601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 976.636978] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:18:00 (1788329880) [ 977.863170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 979.279093] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:18:02 (1788329882) [ 980.481191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 981.770283] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:18:05 (1788329885) [ 982.883307] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 984.514797] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:18:07 (1788329887) [ 985.607407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 986.940283] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:18:10 (1788329890) [ 988.518981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 989.811489] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:18:13 (1788329893) [ 990.081668] LNet: 52760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.635552] Key type .llcrypt unregistered [ 990.641628] Key type ._llcrypt unregistered [ 995.719843] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 995.790799] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 995.799058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 995.806477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 998.261745] Key type ._llcrypt registered [ 998.263797] Key type .llcrypt registered [ 998.422366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.432040] alg: No test for adler32 (adler32-zlib) [ 999.313263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.380549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.421950] LNet: Added LNI 192.168.203.59@tcp [8/512/0/180] [ 999.428782] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 999.432794] LNet: 53077:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.505619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.520403] LNet: 53096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.524675] LNetError: 53096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 999.527943] LNet: Removed LNI 10.1.2.3@tcp [ 999.530375] LNetError: 53096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.548663] LNet: Removed LNI 192.168.203.59@tcp [ 999.562789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.627665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.643225] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 999.705827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.725632] LNet: 53120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.734209] LNetError: 53120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.749980] LNet: Removed LNI 192.168.203.59@tcp [ 999.775534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.843967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.869904] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 999.879572] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.881645] LNet: 53128:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.989986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.009924] LNet: 53146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.015368] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.021137] LNet: Removed LNI 10.1.2.3@tcp [ 1000.023421] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.044263] LNet: Removed LNI 192.168.203.59@tcp [ 1000.068969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.123958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1000.149235] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1000.154158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1000.156425] LNet: 53154:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1000.287899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.309545] LNet: 53180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.314051] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.320402] LNet: Removed LNI 10.1.2.3@tcp [ 1000.322564] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.334892] LNet: Removed LNI 192.168.203.59@tcp [ 1000.351436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.418619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1000.443544] LNet: Added LNI 192.168.203.59@tcp [8/256/0/182] [ 1000.456261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1000.524423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.547912] LNet: 53208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.556319] LNetError: 53208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.566394] LNet: Removed LNI 10.1.2.3@tcp [ 1000.568517] LNetError: 53208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.585883] LNet: Removed LNI 192.168.203.59@tcp [ 1000.609764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.654781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1000.683843] LNet: Added LNI 192.168.203.59@tcp [8/256/0/182] [ 1000.689795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1000.692360] LNet: 53216:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1000.751256] LNet: Removed LNI 10.1.2.3@tcp [ 1000.765636] LNet: Removed LNI 192.168.203.59@tcp [ 1001.282862] Key type .llcrypt unregistered [ 1001.284126] Key type ._llcrypt unregistered [ 1010.684582] Key type ._llcrypt registered [ 1010.687929] Key type .llcrypt registered [ 1011.868948] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 02:18:35 (1788329915) [ 1013.292997] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1014.935340] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:18:38 (1788329918) [ 1015.281139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.290436] alg: No test for adler32 (adler32-zlib) [ 1016.222151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.311990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1016.362288] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1017.046919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.59 [ 1017.063929] LNet: 54477:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1017.074220] LNet: Removed LNI 192.168.203.59@tcp [ 1022.310563] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:18:45 (1788329925) [ 1022.444538] LNet: 55041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.459479] LNetError: 55041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.517799] LNet: Removed LNI 192.168.203.59@tcp [ 1023.111660] Key type .llcrypt unregistered [ 1023.113894] Key type ._llcrypt unregistered [ 1029.494949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1029.769917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1032.630269] Key type ._llcrypt registered [ 1032.636494] Key type .llcrypt registered [ 1032.855460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.877465] alg: No test for adler32 (adler32-zlib) [ 1034.319424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1040.144479] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:19:03 (1788329943) [ 1040.330715] LNet: 55958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.335732] LNetError: 55958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.351771] LNet: Removed LNI 10.1.2.3@tcp [ 1040.842474] Key type .llcrypt unregistered [ 1040.844659] Key type ._llcrypt unregistered [ 1046.418019] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1046.617528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1046.624496] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1046.627193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.300319] Key type ._llcrypt registered [ 1049.302981] Key type .llcrypt registered [ 1049.443785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.458718] alg: No test for adler32 (adler32-zlib) [ 1050.430812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1056.086590] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:19:19 (1788329959) [ 1057.313528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1059.172754] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:19:22 (1788329962) [ 1059.238363] LNet: 57131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.249966] LNetError: 57131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1059.265683] LNet: Removed LNI 10.1.2.3@tcp [ 1059.792913] Key type .llcrypt unregistered [ 1059.796781] Key type ._llcrypt unregistered [ 1064.816519] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1064.991424] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1064.995293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1065.000135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.724086] Key type ._llcrypt registered [ 1067.725761] Key type .llcrypt registered [ 1067.960253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.977090] alg: No test for adler32 (adler32-zlib) [ 1068.870957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.922377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1076.119668] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:19:39 (1788329979) [ 1076.533770] LNet: 58368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.542353] LNetError: 58368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1076.554987] LNet: Removed LNI 10.1.2.3@tcp [ 1077.103852] Key type .llcrypt unregistered [ 1077.113442] Key type ._llcrypt unregistered [ 1082.704572] Key type ._llcrypt registered [ 1082.706615] Key type .llcrypt registered [ 1082.913249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.928431] alg: No test for adler32 (adler32-zlib) [ 1083.870917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.938381] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1090.810933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1091.634478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_interrupt -r 1 [ 1091.702043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_interrupt -r 1 [ 1091.753465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1091.789508] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1092.075750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.123819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1092.128182] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1092.171552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.210613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.255278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.299814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.371083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.420046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.439825] LNet: 58803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.467353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.519833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.570386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.635742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.680433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.711626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.783399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1093.291727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_dropped -r 1 [ 1093.314783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_dropped -r 1 [ 1093.341496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1093.353736] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1093.357366] LNet: Skipped 1 previous similar message [ 1093.564753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.595559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1093.623405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.661242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.700836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.733884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.761573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.803747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.822937] LNet: 58913:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.850205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.907907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.962545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.034356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.068777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.113833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1094.137499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1094.660305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_aborted -r 1 [ 1094.720457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_aborted -r 1 [ 1094.749277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1094.769222] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1094.991881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.030934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1095.068676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.098405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.138353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.176953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.219515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.237440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.249721] LNet: 59022:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.264389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.283545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.307968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.341805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.371234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.397995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.427772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1095.832401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_no_route -r 1 [ 1095.860598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_no_route -r 1 [ 1095.889183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1096.097735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.117456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1096.139880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.168596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.202172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.236145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.260687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.286387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.323660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.358355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.404331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.440353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1096.471870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1096.524815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1096.555699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1097.099285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_timeout -r 1 [ 1097.158241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_timeout -r 1 [ 1097.205366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1097.233766] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1097.236649] LNet: Skipped 2 previous similar messages [ 1097.491231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.523450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1097.549896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.578888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.604734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.639846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.675476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.710938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.729397] LNet: 59241:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.735941] LNet: 59241:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1097.762102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.804662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.846039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.891964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.922875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.955480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.985913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.502284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_error -r 1 [ 1098.536391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_error -r 1 [ 1098.579589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1098.789652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.823910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.863890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.897833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.927839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.977382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.018382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.052876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.090229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.127457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.176810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.222883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.259435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.315548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.354331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.602318] LNet: 59391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.616077] LNetError: 59391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.633154] LNet: Removed LNI 192.168.203.59@tcp [ 1100.031564] Key type .llcrypt unregistered [ 1100.033262] Key type ._llcrypt unregistered [ 1115.223042] Key type ._llcrypt registered [ 1115.226245] Key type .llcrypt registered [ 1116.486895] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:20:19 (1788330019) [ 1117.240270] Key type .llcrypt unregistered [ 1117.244849] Key type ._llcrypt unregistered [ 1123.413988] Key type ._llcrypt registered [ 1123.416543] Key type .llcrypt registered [ 1123.605930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.620369] alg: No test for adler32 (adler32-zlib) [ 1124.572386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.595753] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1132.146611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1141.590653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1141.625908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1141.650469] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1143.384378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_interrupt -r 1 [ 1143.428922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 1 [ 1143.469916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_interrupt -r 1 [ 1143.501397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 1 [ 1143.547228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_interrupt -r 1 [ 1143.587530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 1 [ 1143.613866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_interrupt -r 1 [ 1143.639197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 1 [ 1143.670666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1143.685502] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1143.689575] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1143.913180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.946376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1143.991931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.033516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.074885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.128786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.165912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.199845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.220759] LNet: 60856:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.265683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.324804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.394635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.444162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.498936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.537272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.582691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1145.279744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_dropped -r 1 [ 1145.308681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 1 [ 1145.344781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_dropped -r 1 [ 1145.391504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 1 [ 1145.431405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_dropped -r 1 [ 1145.484632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 1 [ 1145.521527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_dropped -r 1 [ 1145.573626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 1 [ 1145.623809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1145.642656] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1145.650425] LNet: Skipped 2 previous similar messages [ 1145.864887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.907614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.942628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.964689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1146.022123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1146.080546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.176782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1146.204866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1146.228141] LNet: 60977:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1146.256758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1146.303055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1146.385993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.440746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.490751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.547369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.579410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1147.299882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_aborted -r 1 [ 1147.387669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 1 [ 1147.447909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_aborted -r 1 [ 1147.498858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 1 [ 1147.599701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_aborted -r 1 [ 1147.675212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 1 [ 1147.747534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_aborted -r 1 [ 1147.817521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 1 [ 1147.874990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1147.927742] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1147.939861] LNet: Skipped 2 previous similar messages [ 1148.224420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.265938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1148.331778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.367482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.414305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.470746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.545525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.569919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.585931] LNet: 61098:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.613053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.658705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.696114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.736298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.777877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.804615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.838209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.319782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_no_route -r 1 [ 1149.353930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 1 [ 1149.389580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_no_route -r 1 [ 1149.444937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 1 [ 1149.482223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_no_route -r 1 [ 1149.534081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 1 [ 1149.583356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_no_route -r 1 [ 1149.640696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 1 [ 1149.684087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1150.028606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.051679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.079169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.104579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.132437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.156889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.199987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.223902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.252587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.282722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.316633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.364368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.408914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.450068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.490452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1151.064599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_timeout -r 1 [ 1151.120471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 1 [ 1151.159280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_timeout -r 1 [ 1151.207791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 1 [ 1151.264447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_timeout -r 1 [ 1151.304542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 1 [ 1151.325867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_timeout -r 1 [ 1151.371584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 1 [ 1151.416280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1151.440214] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1151.454122] LNet: Skipped 5 previous similar messages [ 1151.706970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.758822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.779738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.800614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.823142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.847951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.867588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.886440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.899385] LNet: 61341:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.904794] LNet: 61341:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1151.916553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.935243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.961903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.982796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.012580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.032985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.053964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1152.434105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_error -r 1 [ 1152.461386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_error -r 1 [ 1152.502839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_error -r 1 [ 1152.564244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_error -r 1 [ 1152.606668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_error -r 1 [ 1152.686702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_error -r 1 [ 1152.770815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_error -r 1 [ 1152.828614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_error -r 1 [ 1152.857703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1153.140704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.167635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.189952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.226417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.254773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.280294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.296776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.332826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.371870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.406500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.429719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.455437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.487725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.523438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.562788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1158.465137] LNet: 61538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.473308] LNetError: 61538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.482604] LNet: Removed LNI 192.168.203.59@tcp [ 1159.529217] LNet: Removed LNI 192.168.203.59@tcp1 [ 1159.905738] Key type .llcrypt unregistered [ 1159.910144] Key type ._llcrypt unregistered [ 1177.015220] Key type ._llcrypt registered [ 1177.017018] Key type .llcrypt registered [ 1178.288405] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:21:21 (1788330081) [ 1179.006566] Key type .llcrypt unregistered [ 1179.009908] Key type ._llcrypt unregistered [ 1184.368881] Key type ._llcrypt registered [ 1184.371874] Key type .llcrypt registered [ 1184.670432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1184.687589] alg: No test for adler32 (adler32-zlib) [ 1185.663937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1185.685269] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1192.475424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1193.031236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_dropped -r 1 [ 1193.058712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_dropped -r 1 [ 1193.087169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1193.113201] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1193.300778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.332646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.358474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.392152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.416900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.449937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.475651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.496831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.510356] LNet: 62783:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.525527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.556283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.589572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.624248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.662444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.693515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.724393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.199764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_error -r 1 [ 1194.270215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_error -r 1 [ 1194.322719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1194.349348] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1194.705967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.761269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1194.835505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.897077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.959612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.035276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.083808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.136879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.150385] LNet: 62893:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1195.178285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.201148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.224364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.245912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.276307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.298761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.325558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.789805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_timeout -r 1 [ 1195.816178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_timeout -r 1 [ 1195.841396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1195.854458] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1196.035811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.072906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.114919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.147142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.181282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.216978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.251144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.279454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.293268] LNet: 63002:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.323297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.353271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.386741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.432428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.466351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.500322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.529305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1196.881678] LNet: 63043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.888760] LNetError: 63043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.903674] LNet: Removed LNI 192.168.203.59@tcp [ 1197.227712] Key type .llcrypt unregistered [ 1197.231512] Key type ._llcrypt unregistered [ 1210.293550] Key type ._llcrypt registered [ 1210.295247] Key type .llcrypt registered [ 1211.441698] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:21:54 (1788330114) [ 1212.172542] Key type .llcrypt unregistered [ 1212.177514] Key type ._llcrypt unregistered [ 1217.218463] Key type ._llcrypt registered [ 1217.220674] Key type .llcrypt registered [ 1217.348396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.360186] alg: No test for adler32 (adler32-zlib) [ 1218.241604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1218.265806] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1226.308621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1237.757384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.850415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1237.893583] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1240.224389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_dropped -r 1 [ 1240.277057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 1 [ 1240.315612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_dropped -r 1 [ 1240.401764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 1 [ 1240.489487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_dropped -r 1 [ 1240.603869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 1 [ 1240.661161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_dropped -r 1 [ 1240.712052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 1 [ 1240.749866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1240.767738] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1240.774052] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 1241.069530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.123993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.153330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.178285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.214941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.246743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.315848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.346123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.362366] LNet: 64505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.374776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.416500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.442123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.464273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.486094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.524652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.563896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1241.588307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.177805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_error -r 1 [ 1242.205449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_error -r 1 [ 1242.238230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_error -r 1 [ 1242.295493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_error -r 1 [ 1242.357537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_error -r 1 [ 1242.418291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_error -r 1 [ 1242.487235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_error -r 1 [ 1242.585393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_error -r 1 [ 1242.684560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1242.732270] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1242.752234] LNet: Skipped 2 previous similar messages [ 1243.164207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.223294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.248732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.277484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.319322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.403891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.484126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.531902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.561026] LNet: 64628:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.599856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.660223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.688196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.713959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.740971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.766365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.827225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.907503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.362364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_timeout -r 1 [ 1244.391935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_timeout -r 1 [ 1244.414138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_timeout -r 1 [ 1244.467930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_timeout -r 1 [ 1244.493368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_timeout -r 1 [ 1244.522802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_timeout -r 1 [ 1244.550579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_timeout -r 1 [ 1244.591401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_timeout -r 1 [ 1244.666125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.159@tcp [ 1244.703712] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1245.012339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.063746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.130831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.181483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.243547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.267901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.295787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.349430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.376485] LNet: 64752:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.402561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.452897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.534595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.621426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.698440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.781630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.842957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.920310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.956679] LNet: 64830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.977059] LNetError: 64830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.989439] LNet: Removed LNI 192.168.203.59@tcp [ 1251.011103] LNet: Removed LNI 192.168.203.59@tcp1 [ 1251.439663] Key type .llcrypt unregistered [ 1251.446748] Key type ._llcrypt unregistered [ 1269.712911] Key type ._llcrypt registered [ 1269.714751] Key type .llcrypt registered [ 1271.191697] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:22:54 (1788330174) [ 1272.707799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1274.609925] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:22:57 (1788330177) [ 1275.502387] Key type .llcrypt unregistered [ 1275.505269] Key type ._llcrypt unregistered [ 1281.782788] Key type ._llcrypt registered [ 1281.786665] Key type .llcrypt registered [ 1282.055869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.087103] alg: No test for adler32 (adler32-zlib) [ 1283.054658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1283.086047] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1290.397785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1291.106910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e network_timeout -r 1 [ 1291.139047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e network_timeout -r 1 [ 1291.181915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1291.201865] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1291.296083] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1291.445718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.481520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.507620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.537178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.567748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.600710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.627034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.663621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.682438] LNet: 66370:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.708551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.737614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.772605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.808439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.834785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.873424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.172988] LNet: 66410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.178897] LNetError: 66410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.193581] LNet: Removed LNI 192.168.203.59@tcp [ 1292.662063] Key type .llcrypt unregistered [ 1292.668699] Key type ._llcrypt unregistered [ 1310.440451] Key type ._llcrypt registered [ 1310.442432] Key type .llcrypt registered [ 1310.710695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1310.740818] alg: No test for adler32 (adler32-zlib) [ 1311.636656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1311.655991] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1319.232259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1329.400642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.443480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1329.485853] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1330.983620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e network_timeout -r 1 [ 1331.029917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e network_timeout -r 1 [ 1331.074308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e network_timeout -r 1 [ 1331.132854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e network_timeout -r 1 [ 1331.180848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e network_timeout -r 1 [ 1331.231200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e network_timeout -r 1 [ 1331.274374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e network_timeout -r 1 [ 1331.326337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e network_timeout -r 1 [ 1331.366237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1331.397277] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1331.656292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.683579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.751307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.804909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.835330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.865541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.895642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.948613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.979949] LNet: 67285:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.000771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.033521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.065647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.087559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.109282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.128314] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1332.134541] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1332.139284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.558606] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1336.990243] LNet: 67359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.010354] LNetError: 67359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.020559] LNet: Removed LNI 192.168.203.59@tcp [ 1337.030269] LNet: Removed LNI 192.168.203.59@tcp1 [ 1337.486837] Key type .llcrypt unregistered [ 1337.493504] Key type ._llcrypt unregistered [ 1353.649194] Key type ._llcrypt registered [ 1353.653260] Key type .llcrypt registered [ 1355.079670] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:24:18 (1788330258) [ 1355.412634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.427513] alg: No test for adler32 (adler32-zlib) [ 1356.383458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.439577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.479449] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1357.329137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1357.360334] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1358.318726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1358.398903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1358.476658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1358.528913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1358.580474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1358.642610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1358.683147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.59@tcp [ 1358.715893] LNetError: 68458:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1359.456895] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1359.458121] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1361.503691] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1362.386757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1365.601723] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1366.082720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1366.124904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.203341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.236803] LNet: 68565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1366.249894] LNetError: 68565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.258993] LNet: Removed LNI 192.168.203.59@tcp [ 1366.289481] LNet: Removed LNI 192.168.203.59@tcp1 [ 1366.357042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.459744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.498621] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1367.264492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.279042] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1367.991828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1368.022270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.066603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1368.105355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1368.142682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1368.178252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1368.224782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1368.264925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.59@tcp [ 1368.295284] LNetError: 68837:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1368.481909] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1370.742965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1374.623422] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1374.636331] LNet: Skipped 1 previous similar message [ 1379.939894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1379.972416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1380.046662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1386.975096] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:24:50 (1788330290) [ 1387.192924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.222332] LNet: 69556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.229734] LNetError: 69556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.234714] LNetError: 69556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1387.240984] LNet: Removed LNI 192.168.203.59@tcp [ 1387.254355] LNet: Removed LNI 192.168.203.59@tcp1 [ 1387.281102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.334469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.362337] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1387.927531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1387.954840] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1388.722255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1388.778343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1388.806179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1388.844125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1388.908407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1388.948377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1388.979991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.59@tcp --health 0 [ 1389.129802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.343161] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.59@tcp [ 1390.367513] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1390.371864] LNet: Skipped 1 previous similar message [ 1405.700044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1405.863095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1405.890756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.915837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.59@tcp --health 500 [ 1406.000631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.649950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.717377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.752964] LNet: 70055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.758867] LNetError: 70055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.764271] LNetError: 70055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1412.772240] LNet: Removed LNI 192.168.203.59@tcp [ 1412.787327] LNet: Removed LNI 192.168.203.59@tcp1 [ 1412.806710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.860946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.879983] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1413.545730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.570258] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1414.318716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1414.379874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.463415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1414.511097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1414.568019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1414.611564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1414.652665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.691000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.59@tcp --health 0 [ 1415.903370] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1415.914380] LNet: Skipped 3 previous similar messages [ 1417.052401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.265210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.304324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1426.370701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.630725] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:25:36 (1788330336) [ 1434.185406] LNet: 71101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.197137] LNetError: 71101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.206401] LNetError: 71101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1434.218224] LNet: Removed LNI 192.168.203.59@tcp [ 1434.236579] LNet: Removed LNI 192.168.203.59@tcp1 [ 1434.838791] Key type .llcrypt unregistered [ 1434.841432] Key type ._llcrypt unregistered [ 1441.435433] Key type ._llcrypt registered [ 1441.437752] Key type .llcrypt registered [ 1441.671906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1441.692454] alg: No test for adler32 (adler32-zlib) [ 1442.617790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1442.646919] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1451.461654] Lustre: DEBUG MARKER: Initial discovery [ 1451.653510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.159@tcp [ 1454.009884] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1454.153545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1454.235233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.159@tcp [ 1454.264963] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1454.303337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1454.354123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.159@tcp -r 1 -m GET -l 3 [ 1454.430049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.159@tcp -r 1 -m GET -e local_error [ 1454.497154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.159@tcp -r 1 -m PUT -l 6 [ 1454.943252] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1456.197557] Lustre: DEBUG MARKER: Force oleg359-client.virtnet to discover 192.168.203.159@tcp (in background) [ 1456.261301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.159@tcp [ 1457.975534] Lustre: DEBUG MARKER: Force oleg359-server to discover 192.168.203.59@tcp [ 1459.364098] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1460.605791] Lustre: DEBUG MARKER: Wait for 71854 [ 1462.252838] Lustre: DEBUG MARKER: Finished wait on 71854 [ 1462.347307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1462.403368] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1462.415845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1462.728512] LNet: 72312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.745756] LNetError: 72312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.797243] LNet: Removed LNI 192.168.203.59@tcp [ 1464.228539] Key type .llcrypt unregistered [ 1464.230938] Key type ._llcrypt unregistered [ 1481.785832] Key type ._llcrypt registered [ 1481.788557] Key type .llcrypt registered [ 1483.021884] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:26:26 (1788330386) [ 1483.702797] Key type .llcrypt unregistered [ 1483.706441] Key type ._llcrypt unregistered [ 1490.494149] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1490.663404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1490.666976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1490.716900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1493.271746] Key type ._llcrypt registered [ 1493.274735] Key type .llcrypt registered [ 1493.444960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1493.463406] alg: No test for adler32 (adler32-zlib) [ 1494.449297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.513105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.568551] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1495.438673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1495.465993] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1504.350267] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:26:47 (1788330407) [ 1504.577451] LNet: 74385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.582803] LNetError: 74385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1504.591608] LNet: Removed LNI 10.1.2.3@tcp [ 1504.604618] LNet: Removed LNI 192.168.203.59@tcp [ 1505.259409] Key type .llcrypt unregistered [ 1505.261314] Key type ._llcrypt unregistered [ 1512.222782] Key type ._llcrypt registered [ 1512.225646] Key type .llcrypt registered [ 1512.413395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1512.437681] alg: No test for adler32 (adler32-zlib) [ 1513.352354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.416164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.433542] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1514.016699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1514.044586] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1514.929064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.59@tcp [ 1516.003773] LNetError: Unexpected error -22 connecting to 192.168.203.59@tcp at host 192.168.203.59:988 [ 1520.823092] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:27:04 (1788330424) [ 1521.045045] LNet: 75515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.054228] LNetError: 75515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1521.068582] LNet: Removed LNI 10.1.2.3@tcp [ 1521.080703] LNet: Removed LNI 192.168.203.59@tcp [ 1521.715523] Key type .llcrypt unregistered [ 1521.718306] Key type ._llcrypt unregistered [ 1528.882215] Key type ._llcrypt registered [ 1528.887280] Key type .llcrypt registered [ 1529.137913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1529.169132] alg: No test for adler32 (adler32-zlib) [ 1530.128365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.214966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1530.248894] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1530.978321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1531.006087] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 1531.797324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.59@tcp1 --nid 192.168.203.59@tcp2 [ 1536.256392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp1 192.168.203.59@tcp2 [ 1536.272194] LNetError: 76348:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.59@tcp1 to 192.168.203.59@tcp2 - Route table may be misconfigured [ 1536.289719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp2 192.168.203.59@tcp1 [ 1542.351564] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:27:25 (1788330445) [ 1542.586962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.614674] LNet: 76922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.623898] LNetError: 76922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.636647] LNet: Removed LNI 192.168.203.59@tcp1 [ 1543.658918] LNet: Removed LNI 192.168.203.59@tcp2 [ 1543.709355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.821447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.848829] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1544.783799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1544.806513] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1545.826850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 1545.890500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.59@tcp [ 1545.939234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp1 192.168.203.59@tcp1 [ 1545.977129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.59@tcp -d 192.168.203.59@tcp -e network_timeout [ 1546.034051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -e network_timeout [ 1546.072460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -e network_timeout [ 1546.106739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -e network_timeout [ 1546.162507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1546.179524] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1546.199951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1546.285460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.380689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1546.849230] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1551.721858] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:27:35 (1788330455) [ 1551.826241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.848550] LNet: 77781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.853761] LNetError: 77781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.863872] LNetError: 77781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1551.870744] LNet: Removed LNI 192.168.203.59@tcp [ 1552.939427] LNet: Removed LNI 192.168.203.59@tcp1 [ 1552.969691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.065623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1553.333686] LNet: 77820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.732793] Key type .llcrypt unregistered [ 1553.736445] Key type ._llcrypt unregistered [ 1564.184442] Key type ._llcrypt registered [ 1564.187702] Key type .llcrypt registered [ 1565.540978] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:27:48 (1788330468) [ 1565.964373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1565.977369] alg: No test for adler32 (adler32-zlib) [ 1566.986183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.477140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1569.876430] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:27:53 (1788330473) [ 1570.136500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.159885] LNet: 78916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.182307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.224039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1570.244984] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1571.009996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1571.042540] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1571.820969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1571.852150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp1 [ 1571.891638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp1 [ 1578.174356] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:28:01 (1788330481) [ 1580.368733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg359-server [ 1582.343915] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:28:05 (1788330485) [ 1584.065485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg359-server [ 1585.881135] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:28:09 (1788330489) [ 1587.419903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg359-server [ 1589.212566] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:28:12 (1788330492) [ 1590.771049] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg359-server [ 1592.305768] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:28:15 (1788330495) [ 1593.967765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg359-server [ 1595.480860] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:28:18 (1788330498) [ 1597.245425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg359-server [ 1598.851446] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:28:22 (1788330502) [ 1600.548406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg359-server [ 1602.101879] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:28:25 (1788330505) [ 1603.987987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg359-server [ 1606.348709] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:28:29 (1788330509) [ 1608.207627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg359-server [ 1609.765100] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:28:33 (1788330513) [ 1610.025566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.046512] LNet: 82794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.051243] LNetError: 82794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.060814] LNet: Removed LNI 192.168.203.59@tcp [ 1611.109545] LNet: Removed LNI 192.168.203.59@tcp1 [ 1611.136433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.212776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.248736] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1611.865583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1611.953662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1612.163969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.190800] LNet: 82941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.203988] LNetError: 82941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.214181] LNetError: 82941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.234327] LNet: Removed LNI 192.168.203.59@tcp [ 1612.258573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.331929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1612.373893] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1613.309230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1613.571508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.605220] LNetError: 83086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.632839] LNet: Removed LNI 192.168.203.59@tcp [ 1614.662842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.705512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.752630] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1615.719626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1615.790556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1616.117375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.156830] LNet: 83234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.162823] LNet: 83234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1616.168686] LNetError: 83234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.188588] LNet: Removed LNI 192.168.203.59@tcp [ 1616.233029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.321575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1616.354831] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1617.325791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1617.561648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.672647] LNet: Removed LNI 192.168.203.59@tcp [ 1618.723579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.800762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.860638] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1619.715380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1619.804124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1620.035708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.092814] LNet: Removed LNI 192.168.203.59@tcp [ 1621.120923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.197330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1621.222845] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1621.875551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1622.127902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.160674] LNet: 83672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.171600] LNet: 83672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1622.184554] LNetError: 83672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.193543] LNetError: 83672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1622.208767] LNet: Removed LNI 192.168.203.59@tcp [ 1622.237659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.300216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.329607] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1622.884627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1622.958744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1623.179768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.229562] LNet: Removed LNI 192.168.203.59@tcp [ 1624.255429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.345849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1624.393498] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1625.275354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1626.680752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.751945] LNet: Removed LNI 192.168.203.59@tcp [ 1627.790598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.873973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.908121] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1628.778966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1628.919424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1629.246805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.314801] LNet: Removed LNI 192.168.203.59@tcp [ 1630.354888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.460229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1630.518146] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1631.301617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1631.590307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.616395] LNet: 84267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.627731] LNet: 84267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1631.640906] LNetError: 84267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.643896] LNetError: 84267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1631.671043] LNet: Removed LNI 192.168.203.59@tcp [ 1631.692246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.757663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1631.786735] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1632.619771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1632.732170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1634.252632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.307871] LNet: Removed LNI 192.168.203.59@tcp [ 1635.373151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.486895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1635.529552] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1636.383191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1637.966662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.078743] LNet: Removed LNI 192.168.203.59@tcp [ 1639.123068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.216334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.257737] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1640.285919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1640.435332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.502928] LNet: Removed LNI 192.168.203.59@tcp [ 1640.544727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.647667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.674397] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1641.479874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1641.551951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.578826] LNet: Removed LNI 192.168.203.59@tcp [ 1641.604970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.668527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1641.786147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.835632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.898271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1641.984840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.022773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.142752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1642.260079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.304520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.397685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1642.446758] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1643.390551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.446166] LNet: Removed LNI 192.168.203.59@tcp [ 1643.488441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.667877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1650.687660] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:29:14 (1788330554) [ 1650.845762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.881405] LNet: 85596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.894407] LNet: 85596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1650.936527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.977274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1650.992977] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1652.292995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.335547] LNetError: 85738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.342216] LNetError: 85738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1652.359929] LNet: Removed LNI 192.168.203.59@tcp231 [ 1652.375900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.491525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1652.541941] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1652.555662] LNet: 85746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.058923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1653.094122] LNet: Removed LNI 192.168.203.59@tcp231 [ 1653.111287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1653.140890] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1654.349275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.419298] LNet: Removed LNI 192.168.203.59@tcp231 [ 1654.478776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.603431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1654.640380] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/0] [ 1654.653472] LNet: 85898:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1655.166055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1655.208507] LNet: Removed LNI 192.168.203.59@tcp231 [ 1655.231817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1655.251486] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/0] [ 1656.579733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.627930] LNet: Removed LNI 192.168.203.59@tcp231 [ 1656.670918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.803802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1656.843949] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/60] [ 1656.855590] LNet: 86050:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.464662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1657.505801] LNet: Removed LNI 192.168.203.59@tcp231 [ 1657.542996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1657.563640] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/60] [ 1658.706877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.782496] LNet: Removed LNI 192.168.203.59@tcp231 [ 1658.820358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.933901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1658.969379] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1658.973204] LNet: 86201:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.495882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1659.603289] LNet: Removed LNI 192.168.203.59@tcp231 [ 1659.631788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1659.660961] LNet: Added LNI 192.168.203.59@tcp231 [8/256/0/180] [ 1667.808341] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:29:31 (1788330571) [ 1667.969663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.011507] LNet: Removed LNI 192.168.203.59@tcp231 [ 1668.029927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.088873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1668.117916] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1668.852914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1668.887423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1668.939067] LNet: Removed LNI 192.168.203.59@tcp232 [ 1668.957195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1668.979870] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1669.729383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1669.776920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1669.820237] LNet: Removed LNI 192.168.203.59@tcp232 [ 1669.845517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1669.890978] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1670.787254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1670.835553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1670.882227] LNet: Removed LNI 192.168.203.59@tcp232 [ 1670.898569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1670.933404] LNet: Added LNI 192.168.203.59@tcp232 [8/256/0/180] [ 1671.872628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1671.934128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1671.973357] LNet: Removed LNI 192.168.203.59@tcp232 [ 1672.002107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1678.303929] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:29:41 (1788330581) [ 1679.423369] Key type .llcrypt unregistered [ 1679.427402] Key type ._llcrypt unregistered [ 1685.671757] Key type ._llcrypt registered [ 1685.675487] Key type .llcrypt registered [ 1685.967281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1685.997908] alg: No test for adler32 (adler32-zlib) [ 1687.022949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1687.090115] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1695.220910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1705.760890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.815923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1705.881445] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1707.751318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_interrupt -r 2 [ 1707.779312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 2 [ 1707.807248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_interrupt -r 2 [ 1707.837902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 2 [ 1707.874510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_interrupt -r 2 [ 1707.914190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_interrupt -r 2 [ 1707.948036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_interrupt -r 2 [ 1707.983990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_interrupt -r 2 [ 1708.030705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1708.056591] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1708.066414] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1708.373185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1708.412733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1708.465964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1708.516093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1708.566911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1708.617489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1708.678697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1708.746639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1708.784033] LNet: 88701:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1708.813732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1708.851832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1708.891941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1708.940965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1708.974637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1709.010403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1709.502191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_dropped -r 2 [ 1709.551629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 2 [ 1709.581785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_dropped -r 2 [ 1709.612110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 2 [ 1709.678633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_dropped -r 2 [ 1709.736465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_dropped -r 2 [ 1709.787286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_dropped -r 2 [ 1709.832615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_dropped -r 2 [ 1709.904523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1709.945099] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1710.280610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1710.322265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1710.353777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1710.398342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1710.448157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1710.494261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1710.544832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1710.605866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1710.657178] LNet: 88821:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1710.689578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1710.729803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1710.790454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1710.870919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1710.927432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1710.963589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1711.567756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_aborted -r 2 [ 1711.605595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 2 [ 1711.637752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_aborted -r 2 [ 1711.672696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 2 [ 1711.715691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_aborted -r 2 [ 1711.778767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_aborted -r 2 [ 1711.812228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_aborted -r 2 [ 1711.842909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_aborted -r 2 [ 1711.873478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1711.895500] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1712.109564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1712.133375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1712.168397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1712.189292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1712.213720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1712.246346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1712.268936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1712.299488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1712.313329] LNet: 88942:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1712.332709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1712.376654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1712.430562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1712.460390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1712.496165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1712.522656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1713.059830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_no_route -r 2 [ 1713.119633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 2 [ 1713.181367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_no_route -r 2 [ 1713.227490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 2 [ 1713.294150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_no_route -r 2 [ 1713.339219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_no_route -r 2 [ 1713.383923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_no_route -r 2 [ 1713.425127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_no_route -r 2 [ 1713.470132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1713.782370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1713.813819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1713.851215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1713.877747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1713.915537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1713.958707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1714.013823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.040376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.069230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.098680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.127920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.162483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.195792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1714.234908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1714.813929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e local_timeout -r 2 [ 1714.842746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 2 [ 1714.874469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e local_timeout -r 2 [ 1714.911118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 2 [ 1714.957054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e local_timeout -r 2 [ 1715.001194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e local_timeout -r 2 [ 1715.059933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e local_timeout -r 2 [ 1715.104355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e local_timeout -r 2 [ 1715.150815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1715.175092] LNet: There was an unexpected network error while writing to 192.168.203.159: rc = -22 [ 1715.180637] LNet: Skipped 1 previous similar message [ 1715.493372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1715.517824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1715.559869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1715.610537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1715.648792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1715.676887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1715.730353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1715.776693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1715.794252] LNet: 89182:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1715.805769] LNet: 89182:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1715.841176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1715.880714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1715.920873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1715.959383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.006513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.062737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1716.713147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp -m GET -e remote_dropped -r 2 [ 1716.801136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 2 [ 1716.884031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp -m GET -e remote_dropped -r 2 [ 1716.947209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 2 [ 1717.026847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp -m GET -e remote_dropped -r 2 [ 1717.124189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.159@tcp1 -m GET -e remote_dropped -r 2 [ 1717.190149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp -m GET -e remote_dropped -r 2 [ 1717.241794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.59@tcp1 -d 192.168.203.59@tcp1 -m GET -e remote_dropped -r 2 [ 1717.318842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.59@tcp 192.168.203.159@tcp [ 1717.350476] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 1717.644186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1717.689452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1717.737483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1717.784623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1717.840512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1717.907989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1717.978253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1718.015772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1718.060247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1718.097831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1718.153142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1718.211848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1718.251909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.300601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1724.543634] LNet: 89377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.550859] LNetError: 89377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.558516] LNet: Removed LNI 192.168.203.59@tcp [ 1724.572398] LNet: Removed LNI 192.168.203.59@tcp1 [ 1725.139585] Key type .llcrypt unregistered [ 1725.144329] Key type ._llcrypt unregistered [ 1744.976188] Key type ._llcrypt registered [ 1744.980329] Key type .llcrypt registered [ 1746.658910] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:30:49 (1788330649) [ 1747.080857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.099993] alg: No test for adler32 (adler32-zlib) [ 1748.124742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.214594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.265869] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1749.349480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1749.401482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.59@tcp [ 1749.429881] LNetError: 90354:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1749.530030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.59@tcp [ 1749.683191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1755.931869] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:30:59 (1788330659) [ 1756.125805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.164899] LNet: 90944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.174740] LNetError: 90944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.191269] LNet: Removed LNI 192.168.203.59@tcp [ 1756.215418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.296058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.325122] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1757.090070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1757.121231] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 1758.370909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.59@tcp [ 1758.397378] LNetError: 91211:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.59@tcp to recovery [ 1759.327358] LNet: 1 local NIs in recovery (showing 1): 192.168.203.59@tcp [ 1759.329441] LNet: There was an unexpected network error while writing to 192.168.203.59: rc = -22 [ 1759.760817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.59@tcp [ 1768.716319] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:31:12 (1788330672) [ 1770.452677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg359-server [ 1772.010034] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:31:15 (1788330675) [ 1773.868424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg359-server [ 1775.495437] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:31:18 (1788330678) [ 1777.213347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg359-server [ 1778.757356] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:31:22 (1788330682) [ 1778.857960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.907802] LNet: 92837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.922811] LNetError: 92837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.932678] LNet: Removed LNI 192.168.203.59@tcp [ 1778.956586] LNet: Removed LNI 192.168.203.59@tcp1 [ 1778.997669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.054555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.083256] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1779.720786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1779.777727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1779.813791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1779.892279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1779.926077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1779.984141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1786.734975] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:31:29 (1788330689) [ 1786.837325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.877544] LNet: 93565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.893427] LNetError: 93565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.918937] LNetError: 93565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1786.944478] LNet: Removed LNI 192.168.203.59@tcp [ 1786.968550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.082573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1787.108404] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1795.410197] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:31:38 (1788330698) [ 1795.541655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.588161] LNet: 94268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.606807] LNetError: 94268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.626831] LNet: Removed LNI 192.168.203.59@tcp [ 1795.659410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.793646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.835050] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1804.755613] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:31:47 (1788330707) [ 1806.754489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1809.359766] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:31:51 (1788330711) [ 1809.971772] LNet: 95312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.981335] LNetError: 95312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.006336] LNet: Removed LNI 192.168.203.59@tcp [ 1810.658652] Key type .llcrypt unregistered [ 1810.660513] Key type ._llcrypt unregistered [ 1818.178434] Key type ._llcrypt registered [ 1818.180224] Key type .llcrypt registered [ 1818.379696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1818.393338] alg: No test for adler32 (adler32-zlib) [ 1819.326072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1819.364119] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1826.254892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 1831.174792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.159@tcp [ 1831.743294] LNet: 95726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.751069] LNetError: 95726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.788276] LNet: Removed LNI 192.168.203.59@tcp [ 1832.246518] Key type .llcrypt unregistered [ 1832.251141] Key type ._llcrypt unregistered [ 1841.894642] Key type ._llcrypt registered [ 1841.897240] Key type .llcrypt registered [ 1843.114779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1844.816937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1846.275225] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:32:29 (1788330749) [ 1847.975332] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1849.568912] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:32:32 (1788330752) [ 1851.090368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg359-server [ 1852.915724] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:32:36 (1788330756) [ 1854.779671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg359-server [ 1856.618622] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:32:39 (1788330759) [ 1857.915399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.929695] alg: No test for adler32 (adler32-zlib) [ 1858.999621] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 1859.204266] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1860.887550] Key type lgssc registered [ 1862.497308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.798922] Key type lgssc unregistered [ 1884.082524] LNet: 98896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.098590] LNetError: 98896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.120405] LNet: Removed LNI 192.168.203.59@tcp [ 1884.844900] Key type .llcrypt unregistered [ 1884.847648] Key type ._llcrypt unregistered [ 1898.714596] Key type ._llcrypt registered [ 1898.718175] Key type .llcrypt registered [ 1900.155341] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:33:23 (1788330803) [ 1900.499671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.513299] alg: No test for adler32 (adler32-zlib) [ 1901.434894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.494822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.531772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.589940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.666801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1907.314664] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:33:30 (1788330810) [ 1907.425836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.447992] LNet: 100303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.478233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.547373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.566577] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1908.331490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.357359] LNet: 100452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.365270] LNetError: 100452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.387900] LNet: Removed LNI 192.168.203.59@tcp [ 1908.424744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.492213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1908.513892] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1908.989284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.018261] LNetError: 100480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.033450] LNet: Removed LNI 192.168.203.59@tcp [ 1909.063108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.124043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1909.150718] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1915.584322] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:33:38 (1788330818) [ 1915.676186] LNet: 101062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.685947] LNet: 101062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1915.700615] LNetError: 101062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.734446] LNet: Removed LNI 192.168.203.59@tcp [ 1916.278492] Key type .llcrypt unregistered [ 1916.285993] Key type ._llcrypt unregistered [ 1921.771248] Key type ._llcrypt registered [ 1921.773768] Key type .llcrypt registered [ 1921.949857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.961661] alg: No test for adler32 (adler32-zlib) [ 1922.902866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.972702] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1922.979943] LNet: *** cfs_fail_loc=e003, val=100*** [ 1923.547277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1923.572446] LNet: *** cfs_fail_loc=e003, val=100*** [ 1923.574745] LNet: Skipped 2 previous similar messages [ 1923.593482] Ping reply received with mapping: [ 1923.595687] found metadata reply NI[192.168.203.59@tcp] [ 1923.600140] 3b cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1923.603096] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1923.606197] 5a 5a 5a 5a 5a 5a 5a 5a [ 1923.746705] LNet: 101465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.754651] LNetError: 101465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.770790] LNet: Removed LNI 192.168.203.59@tcp [ 1924.372738] Key type .llcrypt unregistered [ 1924.375226] Key type ._llcrypt unregistered [ 1936.361693] Key type ._llcrypt registered [ 1936.365715] Key type .llcrypt registered [ 1937.754559] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:34:00 (1788330840) [ 1938.337546] Key type .llcrypt unregistered [ 1938.340707] Key type ._llcrypt unregistered [ 1945.448102] Key type ._llcrypt registered [ 1945.461456] Key type .llcrypt registered [ 1945.769105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1945.790144] alg: No test for adler32 (adler32-zlib) [ 1946.951640] LNetError: 102501:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1947.406098] Key type .llcrypt unregistered [ 1947.409226] Key type ._llcrypt unregistered [ 1952.283744] Key type ._llcrypt registered [ 1952.285730] Key type .llcrypt registered [ 1953.453986] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:34:16 (1788330856) [ 1953.959295] Key type .llcrypt unregistered [ 1953.963156] Key type ._llcrypt unregistered [ 1959.915856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1959.925388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1962.338423] Key type ._llcrypt registered [ 1962.342612] Key type .llcrypt registered [ 1962.492713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1962.505541] alg: No test for adler32 (adler32-zlib) [ 1963.450842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.495719] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1972.307582] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:34:35 (1788330875) [ 1972.527899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.553772] LNet: 104275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.561601] LNetError: 104275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1972.588405] LNet: Removed LNI 10.1.2.3@tcp [ 1972.634383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.307630] Key type .llcrypt unregistered [ 1973.310635] Key type ._llcrypt unregistered [ 1979.782487] Key type ._llcrypt registered [ 1979.790486] Key type .llcrypt registered [ 1979.967831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.979627] alg: No test for adler32 (adler32-zlib) [ 1980.853716] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 1980.898684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 1981.068762] LNet: 104559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.079446] LNetError: 104559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.112448] LNet: Removed LNI 192.168.203.59@tcp [ 1982.021668] Key type .llcrypt unregistered [ 1982.026116] Key type ._llcrypt unregistered [ 1994.026884] Key type ._llcrypt registered [ 1994.028990] Key type .llcrypt registered [ 1995.375291] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:34:58 (1788330898) [ 1996.077304] Key type .llcrypt unregistered [ 1996.080314] Key type ._llcrypt unregistered [ 2002.583541] Key type ._llcrypt registered [ 2002.586696] Key type .llcrypt registered [ 2002.835991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2002.860549] alg: No test for adler32 (adler32-zlib) [ 2010.477428] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:35:13 (1788330913) [ 2010.608575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.639244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.728560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2010.756965] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2011.715496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2011.754534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2011.790598] LNet: Removed LNI 192.168.203.59@tcp [ 2012.042503] LNet: 106297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.401525] Key type .llcrypt unregistered [ 2012.405991] Key type ._llcrypt unregistered [ 2025.742404] Key type ._llcrypt registered [ 2025.748736] Key type .llcrypt registered [ 2027.385283] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:35:30 (1788330930) [ 2028.169266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.189841] alg: No test for adler32 (adler32-zlib) [ 2029.223865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.324680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.350214] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2030.218831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2030.248439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.59@tcp [ 2030.259329] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2030.261840] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2030.264243] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2030, 0, 0 [ 2030.267640] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2030.269485] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2030.272639] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2030.275399] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2030, 0, 0 [ 2030.279304] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2030.281550] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2030.283852] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2030.286345] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2030, 0, 2080 [ 2030.290458] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2030.292466] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2030.295877] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2030.298077] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2030, 0, 2080 [ 2030.300952] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2037.571928] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:35:40 (1788330940) [ 2038.162874] LNet: 107836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.169843] LNetError: 107836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.181961] LNet: Removed LNI 192.168.203.59@tcp [ 2038.789453] Key type .llcrypt unregistered [ 2038.791104] Key type ._llcrypt unregistered [ 2045.873958] Key type ._llcrypt registered [ 2045.877455] Key type .llcrypt registered [ 2046.139828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.153632] alg: No test for adler32 (adler32-zlib) [ 2047.105782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2047.164245] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2054.929820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 2063.033474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.091895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2063.116414] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 2063.968679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2064.072119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2064.077805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2064.082388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2066.431839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2066.459038] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2075.222682] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:36:18 (1788330978) [ 2075.380903] LNet: 109338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.398413] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.413697] LNet: Removed LNI 192.168.203.59@tcp [ 2076.447766] LNet: Removed LNI 192.168.203.59@tcp1 [ 2076.450639] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2076.456019] LNetError: 109338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2076.482679] LNet: Removed LNI 10.1.2.3@tcp99 [ 2077.267943] Key type .llcrypt unregistered [ 2077.269707] Key type ._llcrypt unregistered [ 2083.580401] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2083.582844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2086.060460] Key type ._llcrypt registered [ 2086.063929] Key type .llcrypt registered [ 2086.273761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2086.291054] alg: No test for adler32 (adler32-zlib) [ 2087.178986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.221277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.235211] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2087.902244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2087.927211] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2089.724229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2089.767190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2089.818515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2089.849569] LNetError: 110080:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2089.884336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2089.938883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2090.366259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2090.425209] LNet: 110100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.437543] LNetError: 110100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2090.441946] LNet: Removed LNI 10.1.2.3@tcp [ 2091.499388] LNet: Removed LNI 192.168.203.59@tcp [ 2092.021438] Key type .llcrypt unregistered [ 2092.025068] Key type ._llcrypt unregistered [ 2103.360423] Key type ._llcrypt registered [ 2103.362123] Key type .llcrypt registered [ 2105.101993] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:36:47 (1788331007) [ 2105.502741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.524598] alg: No test for adler32 (adler32-zlib) [ 2106.539324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.631247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.678279] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2107.530387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2107.596770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg359-client@tcp [ 2113.735083] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:36:57 (1788331017) [ 2113.821464] LNet: 111620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.834451] LNetError: 111620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.857524] LNet: Removed LNI 192.168.203.59@tcp [ 2114.504507] Key type .llcrypt unregistered [ 2114.511246] Key type ._llcrypt unregistered [ 2120.556723] Key type ._llcrypt registered [ 2120.565705] Key type .llcrypt registered [ 2120.746907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2120.763485] alg: No test for adler32 (adler32-zlib) [ 2121.652711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2121.671919] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2121.692491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2128.051715] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:37:11 (1788331031) [ 2129.601619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2131.296923] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:37:14 (1788331034) [ 2133.003829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2134.716911] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:37:17 (1788331037) [ 2134.779932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.800400] LNet: 113042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.809751] LNetError: 113042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.829618] LNet: Removed LNI 192.168.203.59@tcp [ 2134.849648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.944879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.970387] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2135.512430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2135.551385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2141.615391] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:37:24 (1788331044) [ 2141.691695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.725445] LNet: 113754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.732612] LNetError: 113754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.753708] LNet: Removed LNI 192.168.203.59@tcp [ 2141.790638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.907384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2141.953231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2142.628386] Key type .llcrypt unregistered [ 2142.634427] Key type ._llcrypt unregistered [ 2154.134725] Key type ._llcrypt registered [ 2154.138232] Key type .llcrypt registered [ 2155.554838] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:37:38 (1788331058) [ 2155.877218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.900133] alg: No test for adler32 (adler32-zlib) [ 2156.940298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.040788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.092071] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2158.208116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2158.317318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2165.062555] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:37:48 (1788331068) [ 2165.284842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.323886] LNet: 115315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.341801] LNetError: 115315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.365231] LNet: Removed LNI 192.168.203.59@tcp [ 2165.435101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.497326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2165.523934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2172.463044] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:37:55 (1788331075) [ 2172.594672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.624608] LNet: 115892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.664781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.766762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2173.015959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2173.050861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2173.101684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2173.489387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2173.521785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2173.613953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2173.639425] LNet: 115929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2173.672208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.798861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2174.139044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2174.168577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2174.220302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2174.585327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2174.617261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2180.269382] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:38:03 (1788331083) [ 2180.369988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.405302] LNet: 116526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.451708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.548072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.587586] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2181.433109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2181.466551] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2182.359995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2182.423859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2182.660621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.681730] LNetError: 116809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.711735] LNet: Removed LNI 192.168.203.59@tcp [ 2183.716746] LNetError: 116809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.742788] LNet: Removed LNI 192.168.203.59@tcp2 [ 2183.763126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.838326] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2183.851869] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2183.857853] LNet: 116815:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2184.395204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2184.621369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.647460] LNet: 116852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.655958] LNet: 116852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2185.695397] LNet: Removed LNI 192.168.203.59@tcp [ 2185.713790] LNet: Removed LNI 192.168.203.59@tcp2 [ 2185.770858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.892720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.936827] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2186.473462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2186.494689] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2187.362456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2187.391232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2187.433273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2187.631740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.673349] LNetError: 117136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.682853] LNetError: 117136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2187.695917] LNet: Removed LNI 192.168.203.59@tcp [ 2188.710723] LNet: Removed LNI 192.168.203.59@tcp2 [ 2188.750874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.847482] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2188.863511] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2188.881068] LNet: 117143:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2189.421606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2189.633663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.664665] LNet: Removed LNI 192.168.203.59@tcp [ 2190.693255] LNet: Removed LNI 192.168.203.59@tcp2 [ 2190.778710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.863445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.917489] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2191.624317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2191.649794] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2192.591832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2192.659447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2192.862546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.882487] LNet: 117460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.886689] LNet: 117460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2192.893537] LNetError: 117460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2192.904519] LNetError: 117460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2192.911866] LNet: Removed LNI 192.168.203.59@tcp [ 2192.929187] LNet: Removed LNI 192.168.203.59@tcp2 [ 2192.967809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.076417] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2193.084186] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2193.090365] LNet: 117466:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2193.666303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2193.860669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.911167] LNet: Removed LNI 192.168.203.59@tcp [ 2194.930620] LNet: Removed LNI 192.168.203.59@tcp2 [ 2194.963745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.085862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2195.122383] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2196.068279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2196.086313] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2196.997909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2197.054702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2197.132926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2197.377728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.434707] LNet: Removed LNI 192.168.203.59@tcp [ 2199.466519] LNet: Removed LNI 192.168.203.59@tcp2 [ 2199.506809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.604422] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2199.617047] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2199.621708] LNet: 117794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2200.088280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2200.274892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.299624] LNet: Removed LNI 192.168.203.59@tcp [ 2201.317886] LNet: Removed LNI 192.168.203.59@tcp2 [ 2201.375646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.449427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2201.481607] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2202.216951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2202.236432] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2203.068429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2203.109393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2203.164563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2203.465820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.482757] LNetError: 118128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.487992] LNetError: 118128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2203.492604] LNet: Removed LNI 192.168.203.59@tcp [ 2203.502828] LNet: Removed LNI 192.168.203.59@tcp2 [ 2203.528451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.579625] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2203.585336] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2204.144823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2204.382567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.409071] LNet: Removed LNI 192.168.203.59@tcp [ 2205.421513] LNet: Removed LNI 192.168.203.59@tcp2 [ 2205.469400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.562811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.616208] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2206.536744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2206.571914] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2207.544327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2207.603322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2207.690568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2207.800592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2208.234888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.311216] LNet: Removed LNI 192.168.203.59@tcp [ 2210.346611] LNet: Removed LNI 192.168.203.59@tcp2 [ 2210.377776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.454657] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2210.460439] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2210.465977] LNet: 118494:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2210.472570] LNet: 118494:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2210.949417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2219.351741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2221.884024] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:38:44 (1788331124) [ 2222.036322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.099730] LNet: 119248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.118054] LNet: 119248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2222.149573] LNetError: 119248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.156035] LNetError: 119248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2222.175607] LNet: Removed LNI 192.168.203.59@tcp [ 2223.218598] LNet: Removed LNI 192.168.203.59@tcp2 [ 2223.284794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.696699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2223.755260] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2225.060817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2225.089614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2226.094144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.110208] LNet: Removed LNI 10.1.2.3@tcp [ 2226.128565] LNet: Removed LNI 192.168.203.59@tcp [ 2226.175725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.296615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2226.344667] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2226.355626] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2227.116606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2227.138630] LNet: Removed LNI 192.168.203.59@tcp [ 2227.153780] LNet: Removed LNI 10.1.2.3@tcp [ 2236.386583] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:38:59 (1788331139) [ 2236.554919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.622585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.715918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2236.743521] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2237.365974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2237.401386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.437501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.468953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.497844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.533639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.573986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.602754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.652899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.697629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.729328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.59@tcp [ 2237.927188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2244.624447] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:39:07 (1788331147) [ 2244.799557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.934627] LNet: Removed LNI 192.168.203.59@tcp [ 2246.013630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.234661] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2246.389983] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2246.402529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2246.415946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2249.054098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2249.131578] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2249.856576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2249.871741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.782284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2251.850291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2251.946091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.59@tcp -m PUT -e local_timeout -r 1 [ 2252.043412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2252.156549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2252.161957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2252.499976] LNetError: 120931:0:(lib-move.c:4801:lnet_parse()) 192.168.203.59@tcp, src 192.168.203.59@tcp: Dropping PUT (error -108 looking up sender) [ 2253.538287] LNet: Removed LNI 10.1.2.3@tcp [ 2254.572982] LNet: Removed LNI 192.168.203.59@tcp [ 2255.097228] Key type .llcrypt unregistered [ 2255.100912] Key type ._llcrypt unregistered [ 2268.709800] Key type ._llcrypt registered [ 2268.714343] Key type .llcrypt registered [ 2270.043578] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:39:33 (1788331173) [ 2270.671778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.687074] alg: No test for adler32 (adler32-zlib) [ 2271.635957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.031721] LNet: 122240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.647480] Key type .llcrypt unregistered [ 2272.650984] Key type ._llcrypt unregistered [ 2279.333111] Key type ._llcrypt registered [ 2279.338753] Key type .llcrypt registered [ 2279.585169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2279.604798] alg: No test for adler32 (adler32-zlib) [ 2280.593170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2280.650649] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2289.132785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 2289.387397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2290.123966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.159@tcp [ 2290.385437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2290.426185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2290.471399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2290.508126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2290.532936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2290.570652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2290.605321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2290.651552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2290.674285] LNet: 122688:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2290.717147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2290.765209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2290.819462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2290.880150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2290.912368] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 2290.921752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2290.965860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2291.847135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2292.081709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2292.119368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2292.152717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2292.184809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2292.219916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2292.248484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2292.310264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2292.344908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2292.358362] LNet: 122805:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2292.380971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2292.427987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2292.482528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2292.531540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2292.561153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2292.599842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2293.290919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.159@tcp [ 2293.527396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2293.572996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2293.618817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2293.663306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2293.694625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2293.745091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2293.781419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2293.830341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2293.854346] LNet: 122922:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2293.879822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2293.940312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2293.980952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2294.014859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2294.050818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2294.081282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2294.841733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2295.107176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2295.144282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2295.193698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2295.233569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2295.269752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2295.307568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.340503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.370022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.418387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.457631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.530932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.604534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.677617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.725510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2296.076945] LNet: 123086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.086207] LNetError: 123086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.104920] LNet: Removed LNI 192.168.203.59@tcp [ 2296.654731] Key type .llcrypt unregistered [ 2296.656411] Key type ._llcrypt unregistered [ 2314.288426] Key type ._llcrypt registered [ 2314.290685] Key type .llcrypt registered [ 2315.664348] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:40:18 (1788331218) [ 2316.098334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.116105] alg: No test for adler32 (adler32-zlib) [ 2317.113855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.635216] LNet: 123971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.353375] Key type .llcrypt unregistered [ 2318.355294] Key type ._llcrypt unregistered [ 2325.290079] Key type ._llcrypt registered [ 2325.294872] Key type .llcrypt registered [ 2325.698412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2325.731889] alg: No test for adler32 (adler32-zlib) [ 2326.677176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2326.698736] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2333.485976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.159@tcp [ 2341.551578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.600618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2341.622236] LNet: Added LNI 192.168.203.59@tcp1 [8/256/0/180] [ 2342.430069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2343.146155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.159@tcp [ 2343.410529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2343.432666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2343.457850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2343.480900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2343.508477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2343.536423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2343.560706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2343.588673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2343.603160] LNet: 124621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2343.623987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2343.657628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2343.687462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2343.714872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2343.738934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2343.767962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2344.334417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2344.531377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2344.562710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2344.586964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2344.621510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2344.653469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2344.682960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2344.741602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2344.776561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2344.794969] LNet: 124738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.810042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2344.850310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2344.891159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2344.938966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2344.975784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2345.013656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2345.675988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.159@tcp [ 2345.909758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2345.933845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2345.964311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2345.998682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2346.037063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2346.081872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2346.130032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2346.177800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2346.192630] LNet: 124855:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2346.210261] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.159@tcp [ 2346.217411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2346.262765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2346.306693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2346.354433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2346.397575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2346.436695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2347.149924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2347.365914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2347.411817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2347.454413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2347.493509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2347.542736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2347.582778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2347.625847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2347.676665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2347.717917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2347.751451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2347.802941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2347.849428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2347.897673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2347.935693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2352.981089] LNet: 125054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.986871] LNetError: 125054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.015252] LNet: Removed LNI 192.168.203.59@tcp [ 2354.019851] LNetError: 125054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.053270] LNet: Removed LNI 192.168.203.59@tcp1 [ 2354.505965] Key type .llcrypt unregistered [ 2354.513833] Key type ._llcrypt unregistered [ 2368.577172] Key type ._llcrypt registered [ 2368.579624] Key type .llcrypt registered [ 2370.057266] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:41:13 (1788331273) [ 2370.448329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.462585] alg: No test for adler32 (adler32-zlib) [ 2371.433460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.486871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.509477] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2372.106642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2372.132888] LNet: Added LNI 192.168.203.59@tcp2 [8/256/0/180] [ 2372.987613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.59@tcp [ 2373.071223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2374.111312] LNet: Removed LNI 192.168.203.59@tcp [ 2380.023325] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:41:23 (1788331283) [ 2381.484409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2383.177418] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:41:26 (1788331286) [ 2391.329934] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:41:34 (1788331294) [ 2398.837050] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:41:42 (1788331302) [ 2406.313593] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:41:49 (1788331309) [ 2413.405127] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:41:56 (1788331316) [ 2420.594734] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:42:03 (1788331323) [ 2428.546662] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:42:11 (1788331331) [ 2438.067323] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:42:20 (1788331340) [ 2446.455764] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:42:29 (1788331349) [ 2455.770840] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:42:39 (1788331359) [ 2465.792525] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:42:48 (1788331368) [ 2474.328496] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:42:57 (1788331377) [ 2481.995084] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:43:05 (1788331385) [ 2489.762677] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:43:12 (1788331392) [ 2498.407117] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:43:21 (1788331401) [ 2506.814625] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:43:29 (1788331409) [ 2515.409098] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:43:38 (1788331418) [ 2523.886375] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:43:47 (1788331427) [ 2524.190558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.218635] LNet: 136750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.223441] LNetError: 136750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2525.289200] LNet: Removed LNI 192.168.203.59@tcp2 [ 2525.314993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2525.521993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2525.552927] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2526.752426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2526.779508] LNet: 136893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.797153] LNetError: 136893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.822809] LNet: Removed LNI 192.168.203.59@tcp [ 2526.843908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2526.894547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2526.935132] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2534.768791] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:43:57 (1788331437) [ 2542.242749] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:44:05 (1788331445) [ 2550.254606] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:44:13 (1788331453) [ 2560.013776] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:44:22 (1788331462) [ 2568.739569] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:44:31 (1788331471) [ 2575.209989] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:44:38 (1788331478) [ 2582.231598] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:44:45 (1788331485) [ 2590.275736] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:44:53 (1788331493) [ 2598.429313] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:45:01 (1788331501) [ 2606.602681] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:45:09 (1788331509) [ 2613.401411] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:45:16 (1788331516) [ 2621.263694] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:45:24 (1788331524) [ 2630.279127] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2344 sec ========== 02:45:32 (1788331532) [ 2630.709565] LNet: 144361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.719559] LNetError: 144361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.752449] LNet: Removed LNI 192.168.203.59@tcp [ 2632.000531] Key type .llcrypt unregistered [ 2632.002762] Key type ._llcrypt unregistered [ 2647.718654] Key type ._llcrypt registered [ 2647.726327] Key type .llcrypt registered [ 2648.189889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2648.208934] alg: No test for adler32 (adler32-zlib) [ 2649.326966] Lustre: Lustre: Build Version: 2.17.57_104_g6cc0f2c [ 2649.560418] LNet: Added LNI 192.168.203.59@tcp [8/256/0/180] [ 2651.327968] Key type lgssc registered [ 2653.042181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2715.412386] Lustre: Mounted lustre-client - version 2.17.57_104_g6cc0f2c [ 2719.851661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2727.533805] Lustre: Unmounted lustre-client [ 2747.753399] Key type lgssc unregistered [ 2748.027421] LNet: 147308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.039803] LNetError: 147308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.066303] LNet: Removed LNI 192.168.203.59@tcp [ 2748.783156] Key type .llcrypt unregistered [ 2748.787735] Key type ._llcrypt unregistered