[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480564306 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003369] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006019] kvm-guest: setup PV IPIs [ 0.009354] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012128] LSM: Security Framework initializing [ 0.013061] Yama: becoming mindful. [ 0.014045] SELinux: Initializing. [ 0.015089] *** VALIDATE selinux *** [ 0.024759] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029811] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031062] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033115] *** VALIDATE tmpfs *** [ 0.035087] *** VALIDATE proc *** [ 0.036254] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.039243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041089] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043032] Spectre V2 : User space: Vulnerable [ 0.044010] Speculative Store Bypass: Vulnerable [ 0.047108] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.049938] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051034] ... version: 2 [ 0.052017] ... bit width: 48 [ 0.053014] ... generic registers: 4 [ 0.054016] ... value mask: 0000ffffffffffff [ 0.055015] ... max period: 00007fffffffffff [ 0.056012] ... fixed-purpose events: 3 [ 0.057012] ... event mask: 000000070000000f [ 0.058337] rcu: Hierarchical SRCU implementation. [ 0.060896] smp: Bringing up secondary CPUs ... [ 0.061785] x86: Booting SMP configuration: [ 0.062040] .... node #0, CPUs: #1 #2 #3 [ 0.066124] smp: Brought up 1 node, 4 CPUs [ 0.068021] smpboot: Max logical packages: 1 [ 0.069022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146027] node 0 deferred pages initialised in 74ms [ 0.148100] devtmpfs: initialized [ 0.149268] x86/mm: Memory block size: 128MB [ 0.152954] gcov: version magic: 0x41383552 [ 0.156030] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158476] pinctrl core: initialized pinctrl subsystem [ 0.160326] [ 0.160939] ************************************************************* [ 0.163013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165012] ** ** [ 0.168014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170011] ** ** [ 0.173014] ** This means that this kernel is built to expose internal ** [ 0.175017] ** IOMMU data structures, which may compromise security on ** [ 0.178016] ** your system. ** [ 0.180068] ** ** [ 0.182011] ** If you see this message and you are not debugging the ** [ 0.185015] ** kernel, report this immediately to your vendor! ** [ 0.187013] ** ** [ 0.190020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192013] ************************************************************* [ 0.195543] NET: Registered protocol family 16 [ 0.197517] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206105] cpuidle: using governor menu [ 0.207823] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210398] PCI: Using configuration type 1 for base access [ 0.212102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224052] cryptd: max_cpu_qlen set to 1000 [ 0.227361] ACPI: Added _OSI(Module Device) [ 0.229018] ACPI: Added _OSI(Processor Device) [ 0.231018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238395] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.244321] ACPI: Interpreter enabled [ 0.245056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247016] ACPI: Using IOAPIC for interrupt routing [ 0.249152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252468] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265203] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278427] acpiphp: Slot [2] registered [ 0.280116] acpiphp: Slot [5] registered [ 0.282215] acpiphp: Slot [6] registered [ 0.283120] acpiphp: Slot [3] registered [ 0.284131] acpiphp: Slot [4] registered [ 0.287114] acpiphp: Slot [7] registered [ 0.289085] acpiphp: Slot [8] registered [ 0.290080] acpiphp: Slot [9] registered [ 0.291083] acpiphp: Slot [10] registered [ 0.292074] acpiphp: Slot [11] registered [ 0.293111] acpiphp: Slot [12] registered [ 0.295133] acpiphp: Slot [13] registered [ 0.296090] acpiphp: Slot [14] registered [ 0.297104] acpiphp: Slot [15] registered [ 0.299119] acpiphp: Slot [16] registered [ 0.301226] acpiphp: Slot [17] registered [ 0.302097] acpiphp: Slot [18] registered [ 0.304095] acpiphp: Slot [19] registered [ 0.305097] acpiphp: Slot [20] registered [ 0.307088] acpiphp: Slot [21] registered [ 0.308109] acpiphp: Slot [22] registered [ 0.310158] acpiphp: Slot [23] registered [ 0.311093] acpiphp: Slot [24] registered [ 0.312187] acpiphp: Slot [25] registered [ 0.314097] acpiphp: Slot [26] registered [ 0.315094] acpiphp: Slot [27] registered [ 0.317104] acpiphp: Slot [28] registered [ 0.318092] acpiphp: Slot [29] registered [ 0.319078] acpiphp: Slot [30] registered [ 0.320092] acpiphp: Slot [31] registered [ 0.322064] PCI host bridge to bus 0000:00 [ 0.324020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334108] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345333] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352645] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.357055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362084] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.365028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368422] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.372397] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.392837] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397092] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402062] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.442174] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484020] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.487395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.489359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.491483] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.494230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.500237] iommu: Default domain type: Passthrough [ 0.502551] SCSI subsystem initialized [ 0.504131] ACPI: bus type USB registered [ 0.505143] usbcore: registered new interface driver usbfs [ 0.507080] usbcore: registered new interface driver hub [ 0.509092] usbcore: registered new device driver usb [ 0.511184] pps_core: LinuxPPS API ver. 1 registered [ 0.513022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.516045] PTP clock support registered [ 0.518137] EDAC MC: Ver: 3.0.0 [ 0.520129] PCI: Using ACPI for IRQ routing [ 0.522732] NetLabel: Initializing [ 0.524011] NetLabel: domain hash size = 128 [ 0.525009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.527075] NetLabel: unlabeled traffic allowed by default [ 0.529337] vgaarb: loaded [ 0.531322] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.532014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.542140] clocksource: Switched to clocksource kvm-clock [ 0.649068] VFS: Disk quotas dquot_6.6.0 [ 0.650686] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.652714] *** VALIDATE ramfs *** [ 0.653857] *** VALIDATE hugetlbfs *** [ 0.654995] pnp: PnP ACPI init [ 0.657095] pnp: PnP ACPI: found 6 devices [ 0.672174] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.676223] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.678512] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.680785] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.682802] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.685011] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.687929] NET: Registered protocol family 2 [ 0.690430] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.695634] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.699444] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.704696] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.707614] TCP: Hash tables configured (established 65536 bind 65536) [ 0.710244] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.712949] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.716087] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.718921] NET: Registered protocol family 1 [ 0.721527] RPC: Registered named UNIX socket transport module. [ 0.723762] RPC: Registered udp transport module. [ 0.725532] RPC: Registered tcp transport module. [ 0.727128] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.729598] NET: Registered protocol family 44 [ 0.731399] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.733515] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.735452] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.737602] PCI: CLS 0 bytes, default 64 [ 0.739631] Unpacking initramfs... [ 2.163788] debug: unmapping init [mem 0xffff901c3cc64000-0xffff901c3ffcffff] [ 2.167723] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.169751] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.172066] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.672481] Initialise system trusted keyrings [ 2.674756] Key type blacklist registered [ 2.676710] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.687064] zbud: loaded [ 2.689978] *** VALIDATE nfs *** [ 2.691393] *** VALIDATE nfs4 *** [ 2.693135] pstore: using deflate compression [ 2.697662] Platform Keyring initialized [ 2.801200] NET: Registered protocol family 38 [ 2.802896] Key type asymmetric registered [ 2.804389] Asymmetric key parser 'x509' registered [ 2.806213] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.809447] io scheduler mq-deadline registered [ 2.811207] io scheduler kyber registered [ 2.813101] io scheduler bfq registered [ 2.815050] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.817853] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.820440] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.823087] ACPI: Power Button [PWRF] [ 2.827813] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.833918] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.843270] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.870659] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.901037] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.907772] Non-volatile memory driver v1.3 [ 2.909887] Linux agpgart interface v0.103 [ 2.941516] virtio_blk virtio1: [vda] 146856 512-byte logical blocks (75.2 MB/71.7 MiB) [ 2.944825] vda: detected capacity change from 0 to 75190272 [ 2.960775] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.964676] vdb: detected capacity change from 0 to 1073741824 [ 2.972043] libphy: Fixed MDIO Bus: probed [ 2.978679] usbcore: registered new interface driver usbserial_generic [ 2.981475] usbserial: USB Serial support registered for generic [ 2.984379] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.988461] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.990437] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.992974] mousedev: PS/2 mouse device common for all mice [ 2.995493] rtc_cmos 00:05: RTC can wake from S4 [ 2.999491] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.000142] rtc_cmos 00:05: registered as rtc0 [ 3.005352] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.005996] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.008763] intel_pstate: CPU model not supported [ 3.015321] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.016515] hid: raw HID events driver (C) Jiri Kosina [ 3.020293] usbcore: registered new interface driver usbhid [ 3.021948] usbhid: USB HID core driver [ 3.023280] drop_monitor: Initializing network drop monitor service [ 3.025096] Initializing XFRM netlink socket [ 3.026943] NET: Registered protocol family 10 [ 3.029793] Segment Routing with IPv6 [ 3.031291] NET: Registered protocol family 17 [ 3.033205] mpls_gso: MPLS GSO support [ 3.039203] RAS: Correctable Errors collector initialized. [ 3.041479] AVX version of gcm_enc/dec engaged. [ 3.042894] AES CTR mode by8 optimization enabled [ 3.115496] sched_clock: Marking stable (3115343123, 0)->(4023211669, -907868546) [ 3.118988] registered taskstats version 1 [ 3.121271] Loading compiled-in X.509 certificates [ 3.123607] zswap: loaded using pool lzo/zbud [ 3.147460] Key type big_key registered [ 3.160164] Key type encrypted registered [ 3.161700] ima: No TPM chip found, activating TPM-bypass! [ 3.163715] ima: Allocated hash algorithm: sha1 [ 3.165555] ima: No architecture policies found [ 3.167330] evm: Initialising EVM extended attributes: [ 3.169225] evm: security.selinux [ 3.170641] evm: security.ima [ 3.171976] evm: security.capability [ 3.173665] evm: HMAC attrs: 0x1 [ 3.175932] rtc_cmos 00:05: setting system clock to 2026-08-27 18:46:39 UTC (1787856399) [ 3.182610] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 3.185320] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 3.194068] Write protecting the kernel read-only data: 28672k [ 3.197736] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.200757] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 3.232562] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.241734] systemd[1]: Detected virtualization kvm. [ 3.243684] systemd[1]: Detected architecture x86-64. [ 3.245599] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.267793] systemd[1]: No hostname configured. [ 3.269281] systemd[1]: Set hostname to . [ 3.271228] random: systemd: uninitialized urandom read (16 bytes read) [ 3.273681] systemd[1]: Initializing machine ID from random generator. [ 3.312428] random: ln: uninitialized urandom read (6 bytes read) [ 3.395224] random: systemd: uninitialized urandom read (16 bytes read) [ 3.398084] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.401990] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.407071] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.957718] device-mapper: uevent: version 1.0.3 [ 3.959835] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.620894] virtio_net virtio0 ens2: renamed from eth0 [ 4.638629] scsi host0: ata_piix [ 4.672438] scsi host1: ata_piix [ 4.673783] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.676347] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.880650] dracut-initqueue[596]: RTNETLINK answers: File exists [ 8.883045] random: fast init done [ 9.678241] random: crng init done [ 9.679505] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.874589] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.912760] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.149207] SELinux: Disabled at runtime. [ 11.205982] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.214493] systemd[1]: Detected virtualization kvm. [ 11.215638] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.671163] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.674394] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.680074] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.685337] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.688703] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.695817] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.700923] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl s[ 11.764462] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tatus proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.115207] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.390528] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.409918] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.515053] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.533690] EDAC sbridge: Ver: 1.1.2 [ 13.448845] Key type dns_resolver registered [ 13.747767] NFS: Registering the id_resolver key type [ 13.749266] Key type id_resolver registered [ 13.750526] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg654-client login: [ 63.084597] libcfs: loading out-of-tree module taints kernel. [ 63.186426] Key type ._llcrypt registered [ 63.193896] Key type .llcrypt registered [ 63.554055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 63.566704] alg: No test for adler32 (adler32-zlib) [ 65.245819] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 66.030224] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 67.863160] Key type lgssc registered [ 69.786027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 193.775492] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec [ 198.725836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.487123] hrtimer: interrupt took 1817129 ns [ 214.119371] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing check_logdir /tmp/testlogs/ [ 217.868406] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing yml_node [ 219.615719] Lustre: lustre-OST0000-osc-ffff901c84cd5800: disconnect after 23s idle [ 220.874058] Lustre: DEBUG MARKER: Client: 2.17.57.81 [ 222.743620] Lustre: DEBUG MARKER: MDS: 2.17.57.81 [ 224.976955] Lustre: DEBUG MARKER: OSS: 2.17.57.81 [ 226.678053] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 14:50:21 EDT 2026 [ 242.163475] Lustre: Unmounted lustre-client [ 259.342814] Key type lgssc unregistered [ 259.705123] LNet: 8435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.709933] LNetError: 8435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.726045] LNet: Removed LNI 192.168.206.54@tcp [ 260.396430] Key type .llcrypt unregistered [ 260.400674] Key type ._llcrypt unregistered [ 284.932450] Key type ._llcrypt registered [ 284.934277] Key type .llcrypt registered [ 285.119275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.140587] alg: No test for adler32 (adler32-zlib) [ 286.022923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 286.066109] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 286.078538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.595283] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 289.865810] LNet: 9323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.888880] LNetError: 9323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.907201] LNet: Removed LNI 192.168.206.54@tcp [ 290.662984] Key type .llcrypt unregistered [ 290.669464] Key type ._llcrypt unregistered [ 297.030842] Key type ._llcrypt registered [ 297.033516] Key type .llcrypt registered [ 298.401957] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:51:33 (1787856693) [ 298.884857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.914650] alg: No test for adler32 (adler32-zlib) [ 299.909221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.022964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.052472] LNet: 9738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.096660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.217557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.252616] LNet: 9745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 306.576736] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:51:41 (1787856701) [ 306.716700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.769996] LNet: 10319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.838425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.948846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 313.709147] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:51:48 (1787856708) [ 313.853640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.882546] LNet: 10904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.926656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.022784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 320.601778] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:51:55 (1787856715) [ 320.694706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.713931] LNet: 11485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.751283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.923811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 326.683691] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:52:01 (1787856721) [ 326.795724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.814755] LNet: 12075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.854663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.903165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 327.365735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 333.111471] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:52:08 (1787856728) [ 333.213419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.250420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.352129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 333.792834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 334.307677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 339.995727] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:52:14 (1787856734) [ 340.123438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.146277] LNet: 13258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.159462] LNet: 13258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 340.209200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.320743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 346.207437] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:52:21 (1787856741) [ 346.439797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.474368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.522806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 346.546579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 346.853451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 346.907668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 347.358152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 347.401465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 347.920533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 347.948598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 348.443982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 348.492135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 348.894777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 348.959577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 349.435563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 349.462829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 349.846239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 349.884331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 350.215339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 350.241022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 350.606876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 350.654695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 351.097747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 351.123767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 351.463923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 351.500547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 351.944311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 351.970685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 356.817510] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:52:32 (1787856752) [ 356.895745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.917401] LNet: 14523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.931246] LNet: 14523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 356.953616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.038472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 357.071914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 362.755983] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:52:37 (1787856757) [ 362.895414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.958901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.047945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 363.090253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 367.842179] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:52:43 (1787856763) [ 368.143304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.192950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.287174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 368.331563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 374.104548] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:52:49 (1787856769) [ 374.311595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.356178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.478458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 374.505266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 381.517174] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:52:56 (1787856776) [ 381.658785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.706758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.812593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 381.867825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 388.222567] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:53:03 (1787856783) [ 388.396453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.422706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.496569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 388.526708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 394.402688] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:53:09 (1787856789) [ 394.692741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.716217] LNet: 18049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.723498] LNet: 18049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 394.744483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.884513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 395.230518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 401.197053] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:53:16 (1787856796) [ 401.336622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.390701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.561688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.811984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 407.084811] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:53:22 (1787856802) [ 407.173419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.233284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.345495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 407.504513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 412.654783] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:53:27 (1787856807) [ 412.776858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.843582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.214161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 413.456782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 418.992252] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:53:34 (1787856814) [ 419.098942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.144900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.218615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 419.260588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 424.849688] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:53:40 (1787856820) [ 424.949434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.000941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.076625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.154544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 430.606527] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:53:45 (1787856825) [ 430.692316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.735966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.858488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 435.598660] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:53:50 (1787856830) [ 435.669210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.712423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.782626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 440.704293] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:53:55 (1787856835) [ 440.840651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.883098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.937909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 441.316506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 446.493846] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:54:01 (1787856841) [ 446.654926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.709343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.798122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 446.841973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 452.090908] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:54:07 (1787856847) [ 452.223753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.250404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.320364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 452.350828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 458.845317] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:54:14 (1787856854) [ 459.022819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.051636] LNet: 24659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.058319] LNet: 24659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 459.087894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.165443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 459.189354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 465.522991] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:54:20 (1787856860) [ 465.610042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.654229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.764467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 465.797762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 471.677368] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:54:26 (1787856866) [ 471.885400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.963331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.042935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 472.118860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 472.142063] LNetError: 25844:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 472.191597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 472.243939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 478.551673] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:54:33 (1787856873) [ 478.587892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.636188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.724782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.IpRuIzUw8y [ 478.776769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.IpRuIzUw8y [ 478.820472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.IpRuIzUw8y [ 484.597527] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:54:39 (1787856879) [ 484.682171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.754470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.846499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 484.867838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 484.914062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 490.511778] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:54:45 (1787856885) [ 490.551631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.610557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.705217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 490.734971] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 492.183228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 492.998646] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 494.911184] Key type lgssc registered [ 495.908738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 499.131962] LNetError: 27603:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.54@tcp99 has no available nets [ 501.946821] Key type lgssc unregistered [ 502.395102] LNetError: 28714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.411590] LNet: Removed LNI 192.168.206.54@tcp [ 503.080263] Key type .llcrypt unregistered [ 503.083303] Key type ._llcrypt unregistered [ 509.606124] Key type ._llcrypt registered [ 509.607749] Key type .llcrypt registered [ 510.650940] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:55:05 (1787856905) [ 511.014386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 511.037749] alg: No test for adler32 (adler32-zlib) [ 511.944489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.011675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 514.991433] LNet: Message forwarding will be enabled in 2s [ 518.047231] LNet: Message forwarding enabled [ 518.428590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 518.451369] LNet: Message forwarding will stop in 2s to 6s [ 523.121927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 523.916541] LNet: Message forwarding disabled [ 525.557151] LNet: Message forwarding will be enabled in 2s [ 529.058266] LNet: Message forwarding enabled [ 531.694436] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:55:26 (1787856926) [ 531.941990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.963254] LNet: 30122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.387391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.519225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 539.302954] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:55:34 (1787856934) [ 539.416054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.463865] LNet: 30699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.512515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.600872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 539.645782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 539.680904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 539.728910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 539.762105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 539.796787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 539.830440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 539.870703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 539.909946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 539.957875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 539.998368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 540.042833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 540.089140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 540.124305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 540.165691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 540.199727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 540.228536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 540.264818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 540.298692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 540.321433] LNetError: 30746:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 545.770463] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:55:40 (1787856940) [ 546.016890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.047317] LNet: 31317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.080668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.162181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.118620] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:55:47 (1787856947) [ 553.865367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg654-server [ 555.503604] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:55:50 (1787856950) [ 555.742494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.791235] LNet: 32247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.823796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.912746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 555.937597] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 556.784268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[55-59]@tcp [ 556.824839] LNet: 32380:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 556.885735] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.58@tcp is being used as a gateway but routing feature is not turned on [ 556.896995] LNetError: 32251:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.58@tcp has gone from up to down [ 557.861736] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 557.874344] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 558.898994] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.57@tcp is being used as a gateway but routing feature is not turned on [ 558.912574] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 560.949415] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 560.975149] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 562.861774] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:55:58 (1787856958) [ 564.453522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg654-server [ 565.031144] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 565.052027] LNetError: 32251:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 565.932024] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:56:01 (1787856961) [ 566.040301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.069739] LNet: 33312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.079522] LNetError: 32262:0:(lib-move.c:4801:lnet_parse()) 192.168.206.59@tcp, src 192.168.206.59@tcp: Dropping ACK (error -108 looking up sender) [ 566.084060] LNetError: 33312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.128084] LNet: Removed LNI 192.168.206.54@tcp [ 566.177412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.228933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 566.260686] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 566.889689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[55-60/2]@tcp [ 566.910107] LNet: 33434:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 566.921839] LNet: 33434:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 566.947522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[55-60/2]@tcp [ 566.963689] LNetError: 33321:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.57@tcp has gone from up to down [ 566.977351] LNetError: 33321:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 571.957912] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:56:07 (1787856967) [ 572.059392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.089931] LNetError: 34026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 572.105984] LNet: Removed LNI 192.168.206.54@tcp [ 572.129805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.211675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 572.254410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.291590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.338507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.379986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.407983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.448057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 572.497424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.535477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.600952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.670172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.708152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.744142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 572.798489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.822845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.867344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.928919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.971488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.035370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 573.131720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.163826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.232995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 573.310271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.362849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.404084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 573.467416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.493384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.539092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 573.583852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.617313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.662798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 573.745449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.799375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.873650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.006338] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:56:14 (1787856974) [ 580.435398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg654-server [ 581.843676] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:56:17 (1787856977) [ 583.284857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg654-server [ 584.754143] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:56:19 (1787856979) [ 584.860780] LNet: 35419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.866590] LNet: 35419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 585.432398] Key type .llcrypt unregistered [ 585.434906] Key type ._llcrypt unregistered [ 591.004528] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 591.162662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 591.169968] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 591.174178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 593.760554] Key type ._llcrypt registered [ 593.766826] Key type .llcrypt registered [ 593.941199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 593.962339] alg: No test for adler32 (adler32-zlib) [ 594.943965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.013338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 595.034705] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 595.623539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 595.640533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 597.319652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 597.335340] LNet: Removed LNI 10.1.2.3@tcp [ 597.442585] LNet: 36151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.447545] LNetError: 36151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.467074] LNet: Removed LNI 192.168.206.54@tcp [ 598.168584] Key type .llcrypt unregistered [ 598.172205] Key type ._llcrypt unregistered [ 608.313505] Key type ._llcrypt registered [ 608.316941] Key type .llcrypt registered [ 609.407884] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:56:44 (1787857004) [ 609.846200] Key type .llcrypt unregistered [ 609.850439] Key type ._llcrypt unregistered [ 615.208411] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 615.408300] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 615.414974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 615.427600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 617.924861] Key type ._llcrypt registered [ 617.931635] Key type .llcrypt registered [ 618.129911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.147895] alg: No test for adler32 (adler32-zlib) [ 619.066498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.111517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.147692] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 619.741990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 619.763705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 622.045813] LNet: 37653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.052609] LNetError: 37653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 622.064064] LNet: Removed LNI 10.1.2.3@tcp [ 622.073225] LNet: Removed LNI 192.168.206.54@tcp [ 622.766463] Key type .llcrypt unregistered [ 622.768088] Key type ._llcrypt unregistered [ 633.805113] Key type ._llcrypt registered [ 633.807286] Key type .llcrypt registered [ 635.010615] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:57:09 (1787857029) [ 635.615253] Key type .llcrypt unregistered [ 635.620289] Key type ._llcrypt unregistered [ 641.447727] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 641.665383] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 641.673632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 641.680844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 644.154449] Key type ._llcrypt registered [ 644.158449] Key type .llcrypt registered [ 644.266564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 644.280612] alg: No test for adler32 (adler32-zlib) [ 645.152180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.238863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 645.268934] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 647.194766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 647.223836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 648.935528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 648.959108] LNet: Removed LNI 10.1.2.3@tcp [ 648.973691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 649.013310] LNet: Removed LNI 10.1.2.31@tcp [ 649.028242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 649.050526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 649.626618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 649.641677] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 650.579149] LNet: 39580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.590924] LNetError: 39580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 650.597793] LNet: Removed LNI 10.1.2.31@tcp [ 650.614257] LNet: Removed LNI 10.1.2.3@tcp [ 651.306512] Key type .llcrypt unregistered [ 651.312282] Key type ._llcrypt unregistered [ 664.037135] Key type ._llcrypt registered [ 664.040590] Key type .llcrypt registered [ 665.324933] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:57:40 (1787857060) [ 665.667692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.679497] alg: No test for adler32 (adler32-zlib) [ 666.569152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.605796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 666.633861] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 667.599665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.624664] LNet: 40536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.637843] LNetError: 40536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.656250] LNet: Removed LNI 192.168.206.54@tcp [ 667.676100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.736637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.54@tcp [ 667.764300] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 668.317554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.54@tcp [ 668.362638] LNet: Removed LNI 192.168.206.54@tcp [ 668.503229] LNet: 40672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.239596] Key type .llcrypt unregistered [ 669.243723] Key type ._llcrypt unregistered [ 681.578872] Key type ._llcrypt registered [ 681.584527] Key type .llcrypt registered [ 683.170575] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:57:57 (1787857077) [ 685.115551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg654-server [ 686.897727] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:58:01 (1787857081) [ 687.429353] Key type .llcrypt unregistered [ 687.431391] Key type ._llcrypt unregistered [ 693.659836] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 693.845237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 693.851718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 693.856139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.375672] Key type ._llcrypt registered [ 696.377826] Key type .llcrypt registered [ 696.588993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.600700] alg: No test for adler32 (adler32-zlib) [ 697.565060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.606175] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 698.379090] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 700.597372] LNet: 42517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.606550] LNetError: 42517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.628454] LNet: Removed LNI 192.168.206.54@tcp1 [ 700.649564] LNet: Removed LNI 10.1.2.3@tcp2 [ 701.392522] Key type .llcrypt unregistered [ 701.397133] Key type ._llcrypt unregistered [ 712.035171] Key type ._llcrypt registered [ 712.037196] Key type .llcrypt registered [ 713.192249] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:58:28 (1787857108) [ 713.596442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 713.623944] alg: No test for adler32 (adler32-zlib) [ 714.623295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.679062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 714.734225] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 715.621140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.634257] LNet: 43491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.639602] LNetError: 43491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.652426] LNet: Removed LNI 192.168.206.54@tcp [ 715.671751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.751209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 715.777813] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 722.660894] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:58:37 (1787857117) [ 722.749979] LNet: 44074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.767570] LNetError: 44074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.794899] LNet: Removed LNI 192.168.206.54@tcp [ 723.502669] Key type .llcrypt unregistered [ 723.505446] Key type ._llcrypt unregistered [ 729.927621] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 730.060123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 730.064533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 730.078237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 732.692776] Key type ._llcrypt registered [ 732.706547] Key type .llcrypt registered [ 732.963870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 732.990194] alg: No test for adler32 (adler32-zlib) [ 733.914470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.023945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.056913] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 734.741297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.764487] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.190320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.203914] LNet: 44829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.207811] LNetError: 44829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 737.211901] LNet: Removed LNI 10.1.2.3@tcp [ 737.222676] LNet: Removed LNI 192.168.206.54@tcp [ 737.242809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.325169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 737.379928] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 737.389266] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.963822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.991887] LNet: 44856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.001876] LNetError: 44856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 738.011074] LNetError: 44856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 738.016769] LNet: Removed LNI 10.1.2.3@tcp [ 738.030296] LNet: Removed LNI 192.168.206.54@tcp [ 738.051491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.131751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 738.159580] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 738.163614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.545108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 738.566295] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 739.622830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.640584] LNet: 45018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.645627] LNetError: 45018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 739.650192] LNetError: 45018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 739.656935] LNet: Removed LNI 10.1.2.3@tcp [ 739.658891] LNet: Removed LNI 192.168.206.54@tcp [ 739.670289] LNet: Removed LNI 192.168.206.54@tcp2 [ 739.683953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.735974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 739.773531] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 739.779049] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 739.784229] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 746.892361] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:59:01 (1787857141) [ 746.956729] LNet: 45609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.967527] LNetError: 45609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.970750] LNetError: 45609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 746.984505] LNet: Removed LNI 10.1.2.3@tcp [ 746.985830] LNet: Removed LNI 192.168.206.54@tcp [ 747.001264] LNet: Removed LNI 192.168.206.54@tcp2 [ 747.659743] Key type .llcrypt unregistered [ 747.663604] Key type ._llcrypt unregistered [ 753.149552] Key type ._llcrypt registered [ 753.152221] Key type .llcrypt registered [ 753.300353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.332150] alg: No test for adler32 (adler32-zlib) [ 754.331822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.407875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.439166] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 755.197801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.218794] LNet: 46013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.229653] LNetError: 46013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.256789] LNet: Removed LNI 192.168.206.54@tcp [ 755.283280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.335543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 755.351600] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 755.803527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.824582] LNet: 46043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.831753] LNetError: 46043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.852833] LNet: Removed LNI 192.168.206.54@tcp [ 755.871320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.956667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 756.013706] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 756.574136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.625447] LNet: Removed LNI 192.168.206.54@tcp [ 756.654549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.719663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 756.741736] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 757.097515] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 757.184332] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 757.196959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 757.213640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.699752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.721465] LNet: 46151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.727476] LNet: 46151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 759.734963] LNetError: 46151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.740587] LNetError: 46151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 759.755737] LNet: Removed LNI 192.168.206.54@tcp [ 759.785841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.861352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 759.887267] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 765.634144] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:59:20 (1787857160) [ 765.785625] LNet: 46745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.801527] LNetError: 46745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.825118] LNet: Removed LNI 192.168.206.54@tcp [ 766.469959] Key type .llcrypt unregistered [ 766.472070] Key type ._llcrypt unregistered [ 771.766135] Key type ._llcrypt registered [ 771.767684] Key type .llcrypt registered [ 771.951818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.985177] alg: No test for adler32 (adler32-zlib) [ 772.967742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.011837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.059483] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 773.854684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.870630] LNet: 47152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.875296] LNetError: 47152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.886958] LNet: Removed LNI 192.168.206.54@tcp [ 773.906478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.958837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 773.981399] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 780.241602] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:59:35 (1787857175) [ 780.456147] LNet: 47745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.461883] LNetError: 47745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.474819] LNet: Removed LNI 192.168.206.54@tcp [ 781.122439] Key type .llcrypt unregistered [ 781.134069] Key type ._llcrypt unregistered [ 787.716945] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.966777] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.972417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.977403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.460278] Key type ._llcrypt registered [ 790.462593] Key type .llcrypt registered [ 790.631802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.644492] alg: No test for adler32 (adler32-zlib) [ 791.598701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.678511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.720072] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 792.607592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.625739] LNet: 48199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.630099] LNetError: 48199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.643628] LNet: Removed LNI 192.168.206.54@tcp [ 792.694696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.787805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 792.825157] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 799.055504] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:59:54 (1787857194) [ 799.237384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.258564] LNet: 48798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.265802] LNetError: 48798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.282991] LNet: Removed LNI 192.168.206.54@tcp [ 799.304347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.405986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 804.519782] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:59:59 (1787857199) [ 804.631172] LNet: 49375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.234658] Key type .llcrypt unregistered [ 805.238421] Key type ._llcrypt unregistered [ 811.564677] Key type ._llcrypt registered [ 811.566279] Key type .llcrypt registered [ 811.803899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 811.832621] alg: No test for adler32 (adler32-zlib) [ 812.773678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.834736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.852730] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 813.623595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.638953] LNet: 49781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.643534] LNetError: 49781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.657300] LNet: Removed LNI 192.168.206.54@tcp [ 813.671410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.728118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 813.766788] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 820.716433] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:00:15 (1787857215) [ 820.836479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.891432] LNet: 50371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.901664] LNetError: 50371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.945658] LNet: Removed LNI 192.168.206.54@tcp [ 820.966895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.034966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 827.189720] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:00:22 (1787857222) [ 828.647913] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 830.190613] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:00:25 (1787857225) [ 831.656641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 833.195507] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:00:28 (1787857228) [ 834.602780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 836.239134] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:00:31 (1787857231) [ 837.512555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 839.010411] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:00:34 (1787857234) [ 840.269795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 841.734831] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:00:36 (1787857236) [ 841.895492] LNet: 52424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.654649] Key type .llcrypt unregistered [ 842.661141] Key type ._llcrypt unregistered [ 848.615165] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.803533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.806912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.810866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.297746] Key type ._llcrypt registered [ 851.300580] Key type .llcrypt registered [ 851.476614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.486385] alg: No test for adler32 (adler32-zlib) [ 852.384140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.445519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 852.472442] LNet: Added LNI 192.168.206.54@tcp [8/512/0/180] [ 852.478662] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 852.484938] LNet: 52750:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 852.554995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.568901] LNet: 52769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.575563] LNetError: 52769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 852.579128] LNet: Removed LNI 10.1.2.3@tcp [ 852.581132] LNetError: 52769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.591885] LNet: Removed LNI 192.168.206.54@tcp [ 852.619334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.682307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 852.706648] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 852.769132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.789315] LNet: 52793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.795439] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.821729] LNet: Removed LNI 192.168.206.54@tcp [ 852.847656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.897214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 852.915801] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 852.919599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.922275] LNet: 52801:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 853.009851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.042624] LNet: 52819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.046906] LNetError: 52819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.060456] LNet: Removed LNI 10.1.2.3@tcp [ 853.064090] LNetError: 52819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.083275] LNet: Removed LNI 192.168.206.54@tcp [ 853.110371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.173630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 853.193202] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 853.199215] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.203437] LNet: 52827:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 853.371530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.397601] LNet: 52853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.406815] LNetError: 52853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.414152] LNet: Removed LNI 10.1.2.3@tcp [ 853.417339] LNetError: 52853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.427302] LNet: Removed LNI 192.168.206.54@tcp [ 853.461638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.518392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 853.534563] LNet: Added LNI 192.168.206.54@tcp [8/256/0/182] [ 853.538040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 853.611377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.627728] LNet: 52881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.632783] LNetError: 52881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.638218] LNet: Removed LNI 10.1.2.3@tcp [ 853.640502] LNetError: 52881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.652088] LNet: Removed LNI 192.168.206.54@tcp [ 853.666301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.710132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 853.726149] LNet: Added LNI 192.168.206.54@tcp [8/256/0/182] [ 853.733176] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 853.734922] LNet: 52889:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 853.799637] LNet: Removed LNI 10.1.2.3@tcp [ 853.810043] LNet: Removed LNI 192.168.206.54@tcp [ 854.301052] Key type .llcrypt unregistered [ 854.304838] Key type ._llcrypt unregistered [ 863.555323] Key type ._llcrypt registered [ 863.559381] Key type .llcrypt registered [ 864.711546] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:00:59 (1787857259) [ 866.315771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 867.557876] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:01:02 (1787857262) [ 867.855434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.865943] alg: No test for adler32 (adler32-zlib) [ 868.735689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.774388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.792235] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 869.316172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.54 [ 869.332798] LNet: 54169:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 869.342730] LNet: Removed LNI 192.168.206.54@tcp [ 873.712840] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:01:08 (1787857268) [ 873.787802] LNet: 54732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.792666] LNetError: 54732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.812643] LNet: Removed LNI 192.168.206.54@tcp [ 874.449333] Key type .llcrypt unregistered [ 874.451396] Key type ._llcrypt unregistered [ 879.347429] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 879.488177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 882.076791] Key type ._llcrypt registered [ 882.079337] Key type .llcrypt registered [ 882.184887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 882.198991] alg: No test for adler32 (adler32-zlib) [ 883.401531] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.496426] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:01:23 (1787857283) [ 888.616786] LNet: 55650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.630256] LNetError: 55650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 888.677751] LNet: Removed LNI 10.1.2.3@tcp [ 889.233634] Key type .llcrypt unregistered [ 889.235396] Key type ._llcrypt unregistered [ 894.558545] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 894.764877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 894.768424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 894.772750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 897.454244] Key type ._llcrypt registered [ 897.456846] Key type .llcrypt registered [ 897.599075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.613914] alg: No test for adler32 (adler32-zlib) [ 898.654732] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.482850] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:01:38 (1787857298) [ 904.532119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 905.876622] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:01:41 (1787857301) [ 905.958622] LNet: 56838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.963491] LNetError: 56838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 905.973251] LNet: Removed LNI 10.1.2.3@tcp [ 906.437589] Key type .llcrypt unregistered [ 906.438820] Key type ._llcrypt unregistered [ 911.482774] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 911.654683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 911.657643] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 911.661026] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 914.315278] Key type ._llcrypt registered [ 914.317437] Key type .llcrypt registered [ 914.460650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.469157] alg: No test for adler32 (adler32-zlib) [ 915.524992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.602497] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.609820] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:01:59 (1787857319) [ 925.192392] LNet: 58079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.198597] LNetError: 58079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 925.222942] LNet: Removed LNI 10.1.2.3@tcp [ 925.955632] Key type .llcrypt unregistered [ 925.962615] Key type ._llcrypt unregistered [ 931.892986] Key type ._llcrypt registered [ 931.895474] Key type .llcrypt registered [ 932.115118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.161382] alg: No test for adler32 (adler32-zlib) [ 933.065152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 933.081840] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 940.270246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 940.969866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_interrupt -r 1 [ 941.014588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_interrupt -r 1 [ 941.044802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 941.072321] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 941.279866] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 941.328728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 941.384152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 941.419703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 941.443416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 941.468163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 941.493145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 941.520452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 941.543660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 941.555190] LNet: 58521:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 941.567350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 941.602056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 941.637862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 941.673414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 941.716824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 941.764307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 941.803354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 942.394501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_dropped -r 1 [ 942.428654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_dropped -r 1 [ 942.482596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 942.517606] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 942.748575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 942.776570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 942.804687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 942.833807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 942.856932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 942.888433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 942.921659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 942.957410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 942.979278] LNet: 58630:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 943.000652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.045696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.093284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.131088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.161135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.241906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.286339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 943.760050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_aborted -r 1 [ 943.791252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_aborted -r 1 [ 943.826158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 943.861372] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 944.080816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.123713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 944.158438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.197305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.233355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.275516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.298289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.317330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.331781] LNet: 58739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 944.348334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.371327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.410272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.440168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.469740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 944.504586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 944.533707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 945.069918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_no_route -r 1 [ 945.102247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_no_route -r 1 [ 945.137200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 945.354323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 945.403684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 945.450610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.495270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.559399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.601755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.649909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.688232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.738381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.791916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.848289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.881436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.911959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.935084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.975478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 946.524143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_timeout -r 1 [ 946.569674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_timeout -r 1 [ 946.621772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 946.650357] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 946.656474] LNet: Skipped 2 previous similar messages [ 946.909443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 946.940844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 946.966522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 946.992176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 947.040704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 947.069548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 947.103277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 947.140060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 947.169293] LNet: 58958:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 947.176355] LNet: 58958:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 947.198805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 947.234980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 947.279522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 947.308659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 947.348319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 947.391305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 947.427157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 947.982252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_error -r 1 [ 948.015910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_error -r 1 [ 948.050393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 948.316086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 948.346644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 948.399543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 948.431350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 948.465696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 948.509911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 948.544913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 948.574335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 948.624352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.655586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.701286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.746273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.774336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 948.839715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 948.875938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 949.187470] LNet: 59112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.198927] LNetError: 59112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.221130] LNet: Removed LNI 192.168.206.54@tcp [ 949.707896] Key type .llcrypt unregistered [ 949.711640] Key type ._llcrypt unregistered [ 964.940156] Key type ._llcrypt registered [ 964.941914] Key type .llcrypt registered [ 966.001834] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:02:41 (1787857361) [ 966.885302] Key type .llcrypt unregistered [ 966.889193] Key type ._llcrypt unregistered [ 972.521215] Key type ._llcrypt registered [ 972.524486] Key type .llcrypt registered [ 972.849132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.874972] alg: No test for adler32 (adler32-zlib) [ 973.818754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 973.845602] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 981.850125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 992.157476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.224602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 992.251187] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 993.659587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_interrupt -r 1 [ 993.717236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 1 [ 993.747925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_interrupt -r 1 [ 993.786399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 1 [ 993.814466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_interrupt -r 1 [ 993.889931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 1 [ 993.960995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_interrupt -r 1 [ 994.027700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 1 [ 994.125777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 994.174424] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 994.177398] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 994.481650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.539437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 994.593877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.668817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.709163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.758154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.804185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.839446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.884095] LNet: 60585:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 994.928453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 995.000785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.052493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.098753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.156669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.190548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.228864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 995.666777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_dropped -r 1 [ 995.700551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 1 [ 995.737160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_dropped -r 1 [ 995.761910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 1 [ 995.793905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_dropped -r 1 [ 995.831072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 1 [ 995.876886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_dropped -r 1 [ 995.934141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 1 [ 995.965447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 995.989183] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 995.993966] LNet: Skipped 2 previous similar messages [ 996.192135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 996.216563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 996.243401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 996.278319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 996.299100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 996.317585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.342593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.371114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.387557] LNet: 60706:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.401556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.428927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.470519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.500489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.535865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.564113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.630854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 997.109950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_aborted -r 1 [ 997.157098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 1 [ 997.189164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_aborted -r 1 [ 997.213486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 1 [ 997.244620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_aborted -r 1 [ 997.265419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 1 [ 997.289645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_aborted -r 1 [ 997.320300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 1 [ 997.372781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 997.390107] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 997.396685] LNet: Skipped 2 previous similar messages [ 997.605695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.627248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 997.656091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.696878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.737980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.766073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.809831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.830244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.853115] LNet: 60828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 997.876941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.924647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.955475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 998.009502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 998.034486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 998.055947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 998.086979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 998.598609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_no_route -r 1 [ 998.638327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 1 [ 998.680843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_no_route -r 1 [ 998.722156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 1 [ 998.750473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_no_route -r 1 [ 998.824204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 1 [ 998.857049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_no_route -r 1 [ 998.907779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 1 [ 998.942301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 999.146869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.177858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 999.204764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 999.222170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 999.242632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 999.265982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.293257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.329763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.371948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.396459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.416332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.461370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.498319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.531127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.552533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1000.108517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_timeout -r 1 [ 1000.173304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 1 [ 1000.221532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_timeout -r 1 [ 1000.272101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 1 [ 1000.327719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_timeout -r 1 [ 1000.391524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 1 [ 1000.440354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_timeout -r 1 [ 1000.469739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 1 [ 1000.523923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1000.549176] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1000.556464] LNet: Skipped 5 previous similar messages [ 1000.775799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1000.804832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.856642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.887727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.953168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.979381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1001.001612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1001.025438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1001.038132] LNet: 61070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1001.043873] LNet: 61070:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1001.060229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1001.091503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1001.117735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1001.138170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1001.159149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1001.187935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1001.236821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1001.801845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_error -r 1 [ 1001.841630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_error -r 1 [ 1001.880477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_error -r 1 [ 1001.931923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_error -r 1 [ 1001.969580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_error -r 1 [ 1001.998170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_error -r 1 [ 1002.040142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_error -r 1 [ 1002.061580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_error -r 1 [ 1002.081105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1002.197726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1002.219978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1002.234171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1002.258128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1002.281481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1002.304472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1002.329751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1002.355392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1002.378439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1002.408705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1002.442920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.489858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.520536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.546664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.575122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1007.686948] LNet: 61272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.699526] LNetError: 61272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.712692] LNet: Removed LNI 192.168.206.54@tcp [ 1007.732150] LNet: Removed LNI 192.168.206.54@tcp1 [ 1008.109681] Key type .llcrypt unregistered [ 1008.112295] Key type ._llcrypt unregistered [ 1026.152314] Key type ._llcrypt registered [ 1026.154657] Key type .llcrypt registered [ 1027.575293] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:03:42 (1787857422) [ 1028.428346] Key type .llcrypt unregistered [ 1028.430069] Key type ._llcrypt unregistered [ 1034.253549] Key type ._llcrypt registered [ 1034.255905] Key type .llcrypt registered [ 1034.478315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.492234] alg: No test for adler32 (adler32-zlib) [ 1035.454405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1035.485412] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1042.533369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1043.282400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_dropped -r 1 [ 1043.312671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_dropped -r 1 [ 1043.348657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1043.375790] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1043.567493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.595164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1043.636993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.671879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.692561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.723385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.768201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.810838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.854271] LNet: 62527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.879528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.922934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.982708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.020427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.087858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.138993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.187793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1044.965704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_error -r 1 [ 1045.021272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_error -r 1 [ 1045.064247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1045.080052] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1045.356991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.391535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1045.422561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.467694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.519649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.588621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.646196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.678729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.703368] LNet: 62636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1045.715382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.757847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.791739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.821850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.852555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.880832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.906788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1046.516827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_timeout -r 1 [ 1046.544112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_timeout -r 1 [ 1046.578201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1046.616892] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1046.874766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.902512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1046.931391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.960880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.006709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.035893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.067277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.095434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.112224] LNet: 62745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.128783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.175778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.220258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.255285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.283512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.309827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.345725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1047.639267] LNet: 62791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.651218] LNetError: 62791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.668832] LNet: Removed LNI 192.168.206.54@tcp [ 1048.160456] Key type .llcrypt unregistered [ 1048.163643] Key type ._llcrypt unregistered [ 1064.012513] Key type ._llcrypt registered [ 1064.017548] Key type .llcrypt registered [ 1065.280492] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:04:20 (1787857460) [ 1066.086361] Key type .llcrypt unregistered [ 1066.088469] Key type ._llcrypt unregistered [ 1071.767541] Key type ._llcrypt registered [ 1071.770227] Key type .llcrypt registered [ 1071.936368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1071.947319] alg: No test for adler32 (adler32-zlib) [ 1072.847809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1072.867530] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1079.573916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1087.939906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1087.987358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1088.022906] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1089.160590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_dropped -r 1 [ 1089.192645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 1 [ 1089.220682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_dropped -r 1 [ 1089.257933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 1 [ 1089.299696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_dropped -r 1 [ 1089.336812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 1 [ 1089.368702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_dropped -r 1 [ 1089.393208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 1 [ 1089.420920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1089.437646] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1089.442371] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 1089.613757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.637848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.664185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.691825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.729690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.782190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.824441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.865295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.883937] LNet: 64277:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.907590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.940232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.971476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1090.018322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1090.072285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.123150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.170737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.210659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.738226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_error -r 1 [ 1090.772737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_error -r 1 [ 1090.812376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_error -r 1 [ 1090.858885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_error -r 1 [ 1090.891833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_error -r 1 [ 1090.931342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_error -r 1 [ 1090.959803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_error -r 1 [ 1090.998336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_error -r 1 [ 1091.040737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1091.064578] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1091.076574] LNet: Skipped 2 previous similar messages [ 1091.380621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.409158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.432706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.470692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.510303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.537187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.582184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.621219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.652864] LNet: 64400:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.676771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.704531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.737986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.762905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.799917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.828856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.860611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1091.889604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.431566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_timeout -r 1 [ 1092.477778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_timeout -r 1 [ 1092.508990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_timeout -r 1 [ 1092.554896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_timeout -r 1 [ 1092.588661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_timeout -r 1 [ 1092.628930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_timeout -r 1 [ 1092.677502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_timeout -r 1 [ 1092.722324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_timeout -r 1 [ 1092.782842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.154@tcp [ 1092.801459] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1093.100876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.140149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1093.196558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.260777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.303534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.373828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.421596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.446236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.466706] LNet: 64523:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.496667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.541762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.587629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.625344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.670332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.706122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.768060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1093.792304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.035465] LNet: 64606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.040841] LNetError: 64606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.049869] LNet: Removed LNI 192.168.206.54@tcp [ 1099.059828] LNet: Removed LNI 192.168.206.54@tcp1 [ 1099.486658] Key type .llcrypt unregistered [ 1099.491828] Key type ._llcrypt unregistered [ 1114.763813] Key type ._llcrypt registered [ 1114.770209] Key type .llcrypt registered [ 1116.034196] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:05:10 (1787857510) [ 1117.411979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1118.941792] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:05:13 (1787857513) [ 1119.951299] Key type .llcrypt unregistered [ 1119.959411] Key type ._llcrypt unregistered [ 1126.508832] Key type ._llcrypt registered [ 1126.510285] Key type .llcrypt registered [ 1126.904456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1126.944707] alg: No test for adler32 (adler32-zlib) [ 1127.892956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1127.928457] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1135.969762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1136.877526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e network_timeout -r 1 [ 1136.908890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e network_timeout -r 1 [ 1136.941505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1136.964297] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1137.184249] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1137.188315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.230562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.275153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.317131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.356916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.422906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.477548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.569289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.601972] LNet: 66155:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.627729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.694727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.783552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.833119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.911586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.962768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.318478] LNet: 66198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.324258] LNetError: 66198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.337672] LNet: Removed LNI 192.168.206.54@tcp [ 1138.697434] Key type .llcrypt unregistered [ 1138.702616] Key type ._llcrypt unregistered [ 1159.435707] Key type ._llcrypt registered [ 1159.438326] Key type .llcrypt registered [ 1159.630334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.661746] alg: No test for adler32 (adler32-zlib) [ 1160.575702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1160.609664] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1167.338355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1176.186747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.239314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1176.262374] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1177.884810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e network_timeout -r 1 [ 1177.944839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e network_timeout -r 1 [ 1177.995481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e network_timeout -r 1 [ 1178.053308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e network_timeout -r 1 [ 1178.097390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e network_timeout -r 1 [ 1178.136728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e network_timeout -r 1 [ 1178.175504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e network_timeout -r 1 [ 1178.220387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e network_timeout -r 1 [ 1178.274279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1178.295580] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1178.551813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.573932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.604724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.633944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.657938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.688557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.717893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.758570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.775755] LNet: 67083:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.799990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.823597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.888811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.929674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.973541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.009288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.737987] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1184.170269] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1184.188783] LNet: 67161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.199543] LNetError: 67161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.212149] LNet: Removed LNI 192.168.206.54@tcp [ 1184.224031] LNet: Removed LNI 192.168.206.54@tcp1 [ 1184.765706] Key type .llcrypt unregistered [ 1184.767292] Key type ._llcrypt unregistered [ 1205.445667] Key type ._llcrypt registered [ 1205.448546] Key type .llcrypt registered [ 1206.807508] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:06:41 (1787857601) [ 1207.534571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1207.548604] alg: No test for adler32 (adler32-zlib) [ 1208.529400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.572943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1208.588244] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1209.697440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1209.737889] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1210.946956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1211.036562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1211.090354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1211.120441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1211.182285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1211.270324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1211.311309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.54@tcp [ 1211.348329] LNetError: 68262:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1211.615401] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1211.615438] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1213.670774] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1215.060356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1217.762851] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1218.676316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1218.707694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.779756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1218.819682] LNet: 68370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.830902] LNetError: 68370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.840804] LNet: Removed LNI 192.168.206.54@tcp [ 1218.853156] LNet: Removed LNI 192.168.206.54@tcp1 [ 1218.878290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.966317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1219.011826] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1219.857045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1219.879893] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1221.003450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1221.088697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.149299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1221.197503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1221.230025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1221.250135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1221.281543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1221.317456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.54@tcp [ 1221.333866] LNetError: 68642:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1221.983595] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1223.664393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1227.103758] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1227.113461] LNet: Skipped 1 previous similar message [ 1231.525339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1231.563422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1231.599553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.002491] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:07:13 (1787857633) [ 1238.051512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.075149] LNet: 69343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.081605] LNetError: 69343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.088319] LNetError: 69343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1238.095215] LNet: Removed LNI 192.168.206.54@tcp [ 1238.104635] LNet: Removed LNI 192.168.206.54@tcp1 [ 1238.127787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.202434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.229784] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1238.852376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.870532] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1239.590467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1239.677672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1239.714804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1239.772789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1239.827978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1239.888353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1239.938194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.54@tcp --health 0 [ 1240.077066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1240.223418] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.54@tcp [ 1240.231886] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1240.238198] LNet: Skipped 1 previous similar message [ 1255.602183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1255.799776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.847031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.892547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.54@tcp --health 500 [ 1256.026598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1262.730900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1262.792889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1262.809317] LNet: 69842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.820472] LNetError: 69842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.826104] LNetError: 69842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1262.831794] LNet: Removed LNI 192.168.206.54@tcp [ 1262.841046] LNet: Removed LNI 192.168.206.54@tcp1 [ 1262.856594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.907988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1262.943616] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1263.715161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1263.738450] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1264.714898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1264.795932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.867713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1264.897605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1264.926618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1264.957493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1264.986320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1265.026868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.54@tcp --health 0 [ 1266.979530] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1266.989832] LNet: Skipped 3 previous similar messages [ 1267.432328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1277.644732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1277.671468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.708783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.417324] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:07:59 (1787857679) [ 1284.866722] LNet: 70899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.876524] LNetError: 70899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.883995] LNetError: 70899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1284.895402] LNet: Removed LNI 192.168.206.54@tcp [ 1284.904975] LNet: Removed LNI 192.168.206.54@tcp1 [ 1285.585330] Key type .llcrypt unregistered [ 1285.587835] Key type ._llcrypt unregistered [ 1291.976500] Key type ._llcrypt registered [ 1291.980377] Key type .llcrypt registered [ 1292.216768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.233505] alg: No test for adler32 (adler32-zlib) [ 1293.201474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1293.231409] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1302.213748] Lustre: DEBUG MARKER: Initial discovery [ 1302.269916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.154@tcp [ 1304.588650] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1304.642324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1304.699703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.154@tcp [ 1304.724429] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1304.748165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1304.788154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.154@tcp -r 1 -m GET -l 3 [ 1304.847227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.154@tcp -r 1 -m GET -e local_error [ 1304.915322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.154@tcp -r 1 -m PUT -l 6 [ 1305.568867] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1306.429115] Lustre: DEBUG MARKER: Force oleg654-client.virtnet to discover 192.168.206.154@tcp (in background) [ 1306.507768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.154@tcp [ 1307.807987] Lustre: DEBUG MARKER: Force oleg654-server to discover 192.168.206.54@tcp [ 1309.602992] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1309.662810] Lustre: DEBUG MARKER: Wait for 71657 [ 1310.867315] Lustre: DEBUG MARKER: Finished wait on 71657 [ 1311.037756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1311.078280] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1311.093434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1311.402444] LNet: 72119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.410868] LNetError: 72119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.500301] LNet: Removed LNI 192.168.206.54@tcp [ 1312.995494] Key type .llcrypt unregistered [ 1312.998751] Key type ._llcrypt unregistered [ 1329.093339] Key type ._llcrypt registered [ 1329.095160] Key type .llcrypt registered [ 1330.240857] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:08:45 (1787857725) [ 1330.907375] Key type .llcrypt unregistered [ 1330.910256] Key type ._llcrypt unregistered [ 1336.755948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1336.973690] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1336.982026] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1336.985891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1339.719329] Key type ._llcrypt registered [ 1339.722987] Key type .llcrypt registered [ 1339.917471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.932910] alg: No test for adler32 (adler32-zlib) [ 1340.859952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.936679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.966183] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1341.576493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1341.591451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1349.262814] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:09:04 (1787857744) [ 1349.398225] LNet: 74198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.408567] LNetError: 74198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1349.418203] LNet: Removed LNI 10.1.2.3@tcp [ 1349.450235] LNet: Removed LNI 192.168.206.54@tcp [ 1350.084897] Key type .llcrypt unregistered [ 1350.089896] Key type ._llcrypt unregistered [ 1356.312822] Key type ._llcrypt registered [ 1356.315482] Key type .llcrypt registered [ 1356.510785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1356.522482] alg: No test for adler32 (adler32-zlib) [ 1357.408111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.451377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.468548] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1358.112786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1358.140564] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1359.264366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.54@tcp [ 1360.351440] LNetError: Unexpected error -22 connecting to 192.168.206.54@tcp at host 192.168.206.54:988 [ 1365.263177] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:09:20 (1787857760) [ 1365.330768] LNet: 75333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.335601] LNetError: 75333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1365.340680] LNet: Removed LNI 10.1.2.3@tcp [ 1365.347338] LNet: Removed LNI 192.168.206.54@tcp [ 1366.068424] Key type .llcrypt unregistered [ 1366.073941] Key type ._llcrypt unregistered [ 1372.831782] Key type ._llcrypt registered [ 1372.833555] Key type .llcrypt registered [ 1373.062494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1373.082513] alg: No test for adler32 (adler32-zlib) [ 1374.042375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.122073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.155579] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1374.936193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1374.957924] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1376.016585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.54@tcp1 --nid 192.168.206.54@tcp2 [ 1380.454778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp1 192.168.206.54@tcp2 [ 1380.480172] LNetError: 76178:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.54@tcp1 to 192.168.206.54@tcp2 - Route table may be misconfigured [ 1380.516359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp2 192.168.206.54@tcp1 [ 1386.429238] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:09:41 (1787857781) [ 1386.686571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.734968] LNet: 76752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.762298] LNetError: 76752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.780142] LNet: Removed LNI 192.168.206.54@tcp1 [ 1387.819900] LNet: Removed LNI 192.168.206.54@tcp2 [ 1387.869397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.936291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.957309] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1388.634191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.662425] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1389.707127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1389.821773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.54@tcp [ 1389.872935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp1 192.168.206.54@tcp1 [ 1389.920936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.54@tcp -d 192.168.206.54@tcp -e network_timeout [ 1389.958705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -e network_timeout [ 1390.012780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -e network_timeout [ 1390.041948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -e network_timeout [ 1390.079528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1390.099729] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1390.124161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1390.218550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1390.305612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1391.012178] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1396.643341] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:09:51 (1787857791) [ 1396.714282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.744879] LNet: 77609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.751321] LNetError: 77609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.756156] LNetError: 77609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1396.760542] LNet: Removed LNI 192.168.206.54@tcp [ 1397.811118] LNet: Removed LNI 192.168.206.54@tcp1 [ 1397.841363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1397.925354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1398.264699] LNet: 77655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.721476] Key type .llcrypt unregistered [ 1398.723847] Key type ._llcrypt unregistered [ 1410.335770] Key type ._llcrypt registered [ 1410.338775] Key type .llcrypt registered [ 1411.719629] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:10:06 (1787857806) [ 1412.189586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1412.213871] alg: No test for adler32 (adler32-zlib) [ 1413.163926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1414.837833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1416.284670] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:10:11 (1787857811) [ 1416.490595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.518968] LNet: 78752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.565943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.615858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1416.646878] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1417.340437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1417.356066] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1418.354674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1418.403593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp1 [ 1418.446239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp1 [ 1424.749973] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:10:19 (1787857819) [ 1426.569709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg654-server [ 1427.946805] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:10:23 (1787857823) [ 1429.542952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg654-server [ 1430.919101] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:10:26 (1787857826) [ 1432.624656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg654-server [ 1433.935409] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:10:29 (1787857829) [ 1435.589199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg654-server [ 1437.186174] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:10:32 (1787857832) [ 1438.847329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg654-server [ 1440.514159] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:10:35 (1787857835) [ 1442.162955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg654-server [ 1443.686946] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:10:38 (1787857838) [ 1445.296943] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg654-server [ 1446.789238] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:10:41 (1787857841) [ 1448.389219] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg654-server [ 1449.750589] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:10:44 (1787857844) [ 1451.209930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg654-server [ 1452.602768] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:10:47 (1787857847) [ 1452.709722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.730786] LNet: 82626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.739493] LNetError: 82626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.747548] LNet: Removed LNI 192.168.206.54@tcp [ 1453.800381] LNet: Removed LNI 192.168.206.54@tcp1 [ 1453.820111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.884660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.908099] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1454.572527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1454.652619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1454.812682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.837282] LNet: 82778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.847410] LNetError: 82778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.852975] LNetError: 82778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1454.866670] LNet: Removed LNI 192.168.206.54@tcp [ 1454.884894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.958890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1454.980785] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1455.637897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1455.832735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.849398] LNetError: 82924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.861818] LNet: Removed LNI 192.168.206.54@tcp [ 1455.889296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.942428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.975787] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1456.529865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1456.616295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1456.795650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.836092] LNet: Removed LNI 192.168.206.54@tcp [ 1457.850874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.898845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1457.924569] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1458.507989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1458.663580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.694338] LNet: 83216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.702458] LNet: 83216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1458.712769] LNetError: 83216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.721541] LNetError: 83216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1458.747186] LNet: Removed LNI 192.168.206.54@tcp [ 1458.766970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.814132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1458.857664] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1459.433479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1459.499850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1459.684484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.775892] LNet: Removed LNI 192.168.206.54@tcp [ 1460.807281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.865336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1460.889343] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1461.523902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1461.759739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1462.821677] LNet: Removed LNI 192.168.206.54@tcp [ 1462.849299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.885771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.902527] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1463.503762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1463.584564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1463.780974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.809782] LNet: 83656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.814596] LNet: 83656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1463.829824] LNetError: 83656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.836548] LNetError: 83656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1464.872370] LNet: Removed LNI 192.168.206.54@tcp [ 1464.889582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.936482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1464.958180] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1465.636512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1465.873144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.916840] LNet: Removed LNI 192.168.206.54@tcp [ 1466.941577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.000400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1467.038792] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1467.627949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1467.694707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1467.912638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.972596] LNet: Removed LNI 192.168.206.54@tcp [ 1469.002919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.059890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1469.084291] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1469.752985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1469.934265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.013712] LNet: Removed LNI 192.168.206.54@tcp [ 1471.039445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.128342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.179632] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1471.995388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1472.082831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1472.414479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.453796] LNet: 84241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.461948] LNet: 84241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1472.481931] LNetError: 84241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.488285] LNetError: 84241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1472.507795] LNet: Removed LNI 192.168.206.54@tcp [ 1472.556932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.589822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1472.606924] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1473.227619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1473.513849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.599225] LNet: Removed LNI 192.168.206.54@tcp [ 1474.621767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.694823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.731370] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1475.402961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1475.473838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.516459] LNet: Removed LNI 192.168.206.54@tcp [ 1475.545585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.599448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1475.626733] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1476.391822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1476.506163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.544354] LNet: Removed LNI 192.168.206.54@tcp [ 1476.584298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.664381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1476.794370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.820617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.870215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1476.961796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.034857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.131655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1477.236920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.266361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.304229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1477.325858] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1477.999773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.022563] LNet: Removed LNI 192.168.206.54@tcp [ 1478.032340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.107180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1484.499795] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:11:19 (1787857879) [ 1484.583693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.636488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.690141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1484.711906] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1485.875862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.925629] LNet: Removed LNI 192.168.206.54@tcp231 [ 1485.943029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.034325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1486.064938] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1486.072179] LNet: 85553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.655859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1486.692790] LNet: Removed LNI 192.168.206.54@tcp231 [ 1486.714972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1486.741045] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1487.760489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.783392] LNet: Removed LNI 192.168.206.54@tcp231 [ 1487.804466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1487.884529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1487.912311] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/0] [ 1487.918261] LNet: 85705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1488.478343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1488.537850] LNet: Removed LNI 192.168.206.54@tcp231 [ 1488.561577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1488.596498] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/0] [ 1489.588593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.605728] LNet: 85849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.610543] LNet: 85849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1489.616561] LNetError: 85849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.621372] LNetError: 85849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1489.635276] LNet: Removed LNI 192.168.206.54@tcp231 [ 1489.658743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.714752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1489.737794] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/60] [ 1489.742352] LNet: 85857:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1490.267412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1490.308096] LNet: Removed LNI 192.168.206.54@tcp231 [ 1490.335327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1490.365700] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/60] [ 1491.536341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.563414] LNet: Removed LNI 192.168.206.54@tcp231 [ 1491.587058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.657108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1491.686615] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1492.130456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1492.158216] LNet: Removed LNI 192.168.206.54@tcp231 [ 1492.174356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1492.209392] LNet: Added LNI 192.168.206.54@tcp231 [8/256/0/180] [ 1498.947822] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:11:33 (1787857893) [ 1499.087597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.132280] LNet: Removed LNI 192.168.206.54@tcp231 [ 1499.158554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.225251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1499.257826] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1499.869997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1499.918848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1499.963423] LNet: Removed LNI 192.168.206.54@tcp232 [ 1499.989975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1500.028988] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1500.885908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1500.918465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1500.947617] LNet: Removed LNI 192.168.206.54@tcp232 [ 1500.963804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1500.978693] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1501.646391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1501.704840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1501.735726] LNet: Removed LNI 192.168.206.54@tcp232 [ 1501.754194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1501.801942] LNet: Added LNI 192.168.206.54@tcp232 [8/256/0/180] [ 1502.720389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1502.751939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1502.782172] LNet: Removed LNI 192.168.206.54@tcp232 [ 1502.811125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1509.060368] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:11:44 (1787857904) [ 1510.272673] Key type .llcrypt unregistered [ 1510.277468] Key type ._llcrypt unregistered [ 1516.374626] Key type ._llcrypt registered [ 1516.376948] Key type .llcrypt registered [ 1516.537858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1516.554546] alg: No test for adler32 (adler32-zlib) [ 1517.456825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1517.490958] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1524.822740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1532.823707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.886247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1532.910390] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1534.204484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_interrupt -r 2 [ 1534.243227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 2 [ 1534.272639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_interrupt -r 2 [ 1534.300992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 2 [ 1534.353451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_interrupt -r 2 [ 1534.411445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_interrupt -r 2 [ 1534.456123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_interrupt -r 2 [ 1534.502940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_interrupt -r 2 [ 1534.545904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1534.566836] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1534.572438] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1534.823131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1534.849383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1534.885740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1534.922566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1534.958591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1534.985872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1535.012727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1535.033356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1535.046228] LNet: 88511:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1535.075418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1535.112812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1535.140371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1535.168913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1535.197835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1535.229415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1535.710989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_dropped -r 2 [ 1535.737157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 2 [ 1535.763783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_dropped -r 2 [ 1535.796764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 2 [ 1535.830132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_dropped -r 2 [ 1535.866528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_dropped -r 2 [ 1535.899863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_dropped -r 2 [ 1535.941602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_dropped -r 2 [ 1535.971286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1535.993266] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1536.226641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1536.266673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1536.300477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1536.333941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1536.384370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1536.410711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1536.449335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1536.478679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1536.492200] LNet: 88632:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1536.511262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1536.547953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1536.585792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1536.620302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1536.682275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1536.717343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1537.245885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_aborted -r 2 [ 1537.283085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 2 [ 1537.309600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_aborted -r 2 [ 1537.341074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 2 [ 1537.372919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_aborted -r 2 [ 1537.404416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_aborted -r 2 [ 1537.439133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_aborted -r 2 [ 1537.467788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_aborted -r 2 [ 1537.507615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1537.532278] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1537.759656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1537.805845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1537.839512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1537.888109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1537.940541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1538.002296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1538.055558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1538.107038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1538.122649] LNet: 88752:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1538.148509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1538.218070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1538.250181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1538.281842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1538.334899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1538.390855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1538.995710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_no_route -r 2 [ 1539.028372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 2 [ 1539.069747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_no_route -r 2 [ 1539.109228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 2 [ 1539.150752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_no_route -r 2 [ 1539.187506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_no_route -r 2 [ 1539.239407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_no_route -r 2 [ 1539.298770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_no_route -r 2 [ 1539.352280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1539.699805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1539.733426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1539.775805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1539.812756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1539.838568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1539.867770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1539.902237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1539.933878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1539.965828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1540.000780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1540.050516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1540.077229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1540.130034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1540.166556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1540.714962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e local_timeout -r 2 [ 1540.767043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 2 [ 1540.838471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e local_timeout -r 2 [ 1540.880529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 2 [ 1540.942613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e local_timeout -r 2 [ 1540.996213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e local_timeout -r 2 [ 1541.045510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e local_timeout -r 2 [ 1541.092937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e local_timeout -r 2 [ 1541.122640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1541.156704] LNet: There was an unexpected network error while writing to 192.168.206.154: rc = -22 [ 1541.158876] LNet: Skipped 1 previous similar message [ 1541.391712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1541.426527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1541.456758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1541.496632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1541.527599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1541.576785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1541.635655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1541.688906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1541.723345] LNet: 88992:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1541.729736] LNet: 88992:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1541.754045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1541.791422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1541.840781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1541.897490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1541.944579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1542.009621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1542.539638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp -m GET -e remote_dropped -r 2 [ 1542.576822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 2 [ 1542.623560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp -m GET -e remote_dropped -r 2 [ 1542.680454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 2 [ 1542.736820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp -m GET -e remote_dropped -r 2 [ 1542.806946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.54@tcp1 -d 192.168.206.154@tcp1 -m GET -e remote_dropped -r 2 [ 1542.853308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp -m GET -e remote_dropped -r 2 [ 1542.887363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.54@tcp1 -d 192.168.206.54@tcp1 -m GET -e remote_dropped -r 2 [ 1542.927511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.54@tcp 192.168.206.154@tcp [ 1542.959612] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 1543.201996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1543.249161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1543.331411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1543.356251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1543.412875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1543.467908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1543.497825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1543.535719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1543.557602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1543.583316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1543.612503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1543.643443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1543.683792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1543.711652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1549.600514] LNet: 89191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.613804] LNetError: 89191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.622628] LNet: Removed LNI 192.168.206.54@tcp [ 1549.631934] LNet: Removed LNI 192.168.206.54@tcp1 [ 1549.989642] Key type .llcrypt unregistered [ 1549.993508] Key type ._llcrypt unregistered [ 1567.916894] Key type ._llcrypt registered [ 1567.919934] Key type .llcrypt registered [ 1569.403534] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:12:44 (1787857964) [ 1569.914414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.928386] alg: No test for adler32 (adler32-zlib) [ 1570.885894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.959954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1570.980903] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1571.859319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1571.886949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.54@tcp [ 1571.906208] LNetError: 90169:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1571.939320] LNet: 1 local NIs in recovery (showing 1): 192.168.206.54@tcp [ 1572.026980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.54@tcp [ 1572.187434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.569692] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:12:53 (1787857973) [ 1578.768336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.805293] LNet: 90759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.815383] LNetError: 90759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.833935] LNet: Removed LNI 192.168.206.54@tcp [ 1578.855418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.957964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.977181] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1579.577328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1579.595119] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1580.735413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.54@tcp [ 1580.773785] LNetError: 91025:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.54@tcp to recovery [ 1580.961658] LNet: There was an unexpected network error while writing to 192.168.206.54: rc = -22 [ 1582.046682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.54@tcp [ 1589.756434] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:13:04 (1787857984) [ 1591.492056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg654-server [ 1593.076584] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:13:08 (1787857988) [ 1594.774449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg654-server [ 1596.636505] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:13:11 (1787857991) [ 1598.486315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg654-server [ 1600.264487] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:13:15 (1787857995) [ 1600.402813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.435937] LNet: 92642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.456335] LNetError: 92642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1600.463950] LNet: Removed LNI 192.168.206.54@tcp [ 1600.512473] LNet: Removed LNI 192.168.206.54@tcp1 [ 1600.541044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.589842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.607578] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1601.262793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1601.311593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1601.359558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1601.412879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1601.451920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1601.486680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1608.024642] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:13:22 (1787858002) [ 1608.171809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.210650] LNet: 93368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.221330] LNetError: 93368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.225492] LNetError: 93368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1608.250547] LNet: Removed LNI 192.168.206.54@tcp [ 1608.288072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.373954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1608.402404] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1615.554956] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:13:30 (1787858010) [ 1615.670819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.693326] LNet: 94071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.713328] LNetError: 94071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.760712] LNet: Removed LNI 192.168.206.54@tcp [ 1615.782520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.828828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.852186] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1623.029442] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:13:37 (1787858017) [ 1624.573134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1626.003715] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:13:41 (1787858021) [ 1626.460730] LNet: 95111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.465861] LNetError: 95111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.478298] LNet: Removed LNI 192.168.206.54@tcp [ 1626.983076] Key type .llcrypt unregistered [ 1626.987830] Key type ._llcrypt unregistered [ 1632.991868] Key type ._llcrypt registered [ 1632.993306] Key type .llcrypt registered [ 1633.259399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1633.271986] alg: No test for adler32 (adler32-zlib) [ 1634.148449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1634.171062] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1641.773656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1647.123528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.154@tcp [ 1647.582500] LNet: 95536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.587968] LNetError: 95536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.604856] LNet: Removed LNI 192.168.206.54@tcp [ 1648.057493] Key type .llcrypt unregistered [ 1648.066980] Key type ._llcrypt unregistered [ 1659.979985] Key type ._llcrypt registered [ 1659.982222] Key type .llcrypt registered [ 1661.472227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1662.985963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1664.524068] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:14:19 (1787858059) [ 1666.143685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1668.032606] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:14:22 (1787858062) [ 1670.010426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg654-server [ 1671.543372] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:14:26 (1787858066) [ 1673.220537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg654-server [ 1675.024785] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:14:30 (1787858070) [ 1676.499563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.551742] alg: No test for adler32 (adler32-zlib) [ 1677.728724] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 1678.006044] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1679.695180] Key type lgssc registered [ 1680.825469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1698.898076] Key type lgssc unregistered [ 1699.132819] LNet: 98711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.140893] LNetError: 98711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.160553] LNet: Removed LNI 192.168.206.54@tcp [ 1699.678716] Key type .llcrypt unregistered [ 1699.682624] Key type ._llcrypt unregistered [ 1712.928530] Key type ._llcrypt registered [ 1712.931813] Key type .llcrypt registered [ 1714.067827] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:15:09 (1787858109) [ 1714.378595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.409706] alg: No test for adler32 (adler32-zlib) [ 1715.277407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.334595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1715.384839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1715.426743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1715.488984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1721.010509] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:15:16 (1787858116) [ 1721.173439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.195206] LNet: 100127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.224202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.283638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.309278] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1722.099755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.115369] LNet: 100271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.123171] LNetError: 100271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.136665] LNet: Removed LNI 192.168.206.54@tcp [ 1722.152816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.215430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1722.251699] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1722.771328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.786782] LNetError: 100299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.802709] LNet: Removed LNI 192.168.206.54@tcp [ 1722.824253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.891215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1722.919072] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1728.834424] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:15:24 (1787858124) [ 1728.949960] LNet: 100884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.958444] LNet: 100884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1728.971773] LNetError: 100884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.010158] LNet: Removed LNI 192.168.206.54@tcp [ 1729.612863] Key type .llcrypt unregistered [ 1729.615670] Key type ._llcrypt unregistered [ 1735.183659] Key type ._llcrypt registered [ 1735.186737] Key type .llcrypt registered [ 1735.444732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1735.475094] alg: No test for adler32 (adler32-zlib) [ 1736.362563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.437469] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1736.441390] LNet: *** cfs_fail_loc=e003, val=100*** [ 1737.262268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1737.283533] LNet: *** cfs_fail_loc=e003, val=100*** [ 1737.287127] LNet: Skipped 2 previous similar messages [ 1737.293497] Ping reply received with mapping: [ 1737.296821] found metadata reply NI[192.168.206.54@tcp] [ 1737.299315] 36 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1737.301314] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1737.303823] 5a 5a 5a 5a 5a 5a 5a 5a [ 1737.443876] LNet: 101290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.449475] LNetError: 101290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.466599] LNet: Removed LNI 192.168.206.54@tcp [ 1738.179864] Key type .llcrypt unregistered [ 1738.183122] Key type ._llcrypt unregistered [ 1748.491200] Key type ._llcrypt registered [ 1748.493695] Key type .llcrypt registered [ 1749.512671] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:15:44 (1787858144) [ 1749.975368] Key type .llcrypt unregistered [ 1749.980270] Key type ._llcrypt unregistered [ 1756.122711] Key type ._llcrypt registered [ 1756.125742] Key type .llcrypt registered [ 1756.252735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1756.271816] alg: No test for adler32 (adler32-zlib) [ 1757.463068] LNetError: 102346:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1757.861437] Key type .llcrypt unregistered [ 1757.867717] Key type ._llcrypt unregistered [ 1762.613652] Key type ._llcrypt registered [ 1762.616700] Key type .llcrypt registered [ 1763.823562] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:15:58 (1787858158) [ 1764.301401] Key type .llcrypt unregistered [ 1764.306832] Key type ._llcrypt unregistered [ 1770.327656] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1770.341704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1772.785639] Key type ._llcrypt registered [ 1772.788342] Key type .llcrypt registered [ 1772.977365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1773.002308] alg: No test for adler32 (adler32-zlib) [ 1773.969281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.043763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1782.839453] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:16:17 (1787858177) [ 1782.957842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.001255] LNet: 104128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.007192] LNetError: 104128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1783.035305] LNet: Removed LNI 10.1.2.3@tcp [ 1783.061922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.772687] Key type .llcrypt unregistered [ 1783.776219] Key type ._llcrypt unregistered [ 1789.533307] Key type ._llcrypt registered [ 1789.537150] Key type .llcrypt registered [ 1789.719203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1789.736978] alg: No test for adler32 (adler32-zlib) [ 1790.639083] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1790.691206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1790.800491] LNet: 104428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.805851] LNetError: 104428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.820252] LNet: Removed LNI 192.168.206.54@tcp [ 1791.346452] Key type .llcrypt unregistered [ 1791.349810] Key type ._llcrypt unregistered [ 1802.008960] Key type ._llcrypt registered [ 1802.012939] Key type .llcrypt registered [ 1803.391437] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:16:38 (1787858198) [ 1804.147750] Key type .llcrypt unregistered [ 1804.155986] Key type ._llcrypt unregistered [ 1811.686376] Key type ._llcrypt registered [ 1811.688690] Key type .llcrypt registered [ 1811.855262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1811.883436] alg: No test for adler32 (adler32-zlib) [ 1818.268202] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:16:53 (1787858213) [ 1818.372525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.396130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.456976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.493613] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1819.153215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.184745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1819.226569] LNet: Removed LNI 192.168.206.54@tcp [ 1819.479894] LNet: 106185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.877432] Key type .llcrypt unregistered [ 1819.879624] Key type ._llcrypt unregistered [ 1831.177392] Key type ._llcrypt registered [ 1831.179958] Key type .llcrypt registered [ 1832.626275] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:17:07 (1787858227) [ 1833.001235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.015315] alg: No test for adler32 (adler32-zlib) [ 1833.923720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.977443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.004340] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1834.736535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1834.763384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.54@tcp [ 1834.792497] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1834.798929] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.810174] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 0 [ 1834.821695] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1834.825044] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1834.829455] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.833965] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 0 [ 1834.839390] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1834.846514] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1834.853517] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.861745] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 1884 [ 1834.866904] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1834.869636] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1834.875457] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.878298] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 1884 [ 1834.883901] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1841.138656] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:17:16 (1787858236) [ 1841.629691] LNet: 107725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.635847] LNetError: 107725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.650166] LNet: Removed LNI 192.168.206.54@tcp [ 1842.313391] Key type .llcrypt unregistered [ 1842.319548] Key type ._llcrypt unregistered [ 1848.658165] Key type ._llcrypt registered [ 1848.660799] Key type .llcrypt registered [ 1848.835505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1848.844388] alg: No test for adler32 (adler32-zlib) [ 1849.760264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1849.813924] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1856.695446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 1865.077831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.127794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1865.147430] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 1866.000982] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1866.070644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1866.073866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1866.077524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1868.566824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1868.586495] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1878.591077] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:17:53 (1787858273) [ 1878.743018] LNet: 109233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.751554] LNetError: 109233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.767040] LNet: Removed LNI 192.168.206.54@tcp [ 1879.775255] LNet: Removed LNI 192.168.206.54@tcp1 [ 1879.781470] LNetError: 109233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1879.788021] LNetError: 109233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1879.824294] LNet: Removed LNI 10.1.2.3@tcp99 [ 1880.479995] Key type .llcrypt unregistered [ 1880.482129] Key type ._llcrypt unregistered [ 1886.241783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1886.244991] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1888.742303] Key type ._llcrypt registered [ 1888.745150] Key type .llcrypt registered [ 1888.947099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1888.960237] alg: No test for adler32 (adler32-zlib) [ 1889.890669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.971927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.998318] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1890.836910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1890.871133] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1892.938338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1892.979921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1893.018971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1893.041541] LNetError: 109994:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1893.067652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1893.103359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1893.505163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1893.625650] LNet: 110014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.639497] LNetError: 110014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1893.653428] LNet: Removed LNI 10.1.2.3@tcp [ 1894.693306] LNet: Removed LNI 192.168.206.54@tcp [ 1895.458560] Key type .llcrypt unregistered [ 1895.460628] Key type ._llcrypt unregistered [ 1906.337905] Key type ._llcrypt registered [ 1906.339748] Key type .llcrypt registered [ 1907.464839] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:18:22 (1787858302) [ 1907.910307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.929993] alg: No test for adler32 (adler32-zlib) [ 1908.850792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.922775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.957429] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1909.660207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 1909.728118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg654-client@tcp [ 1915.250055] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:18:30 (1787858310) [ 1915.335550] LNet: 111541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.338261] LNetError: 111541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.357259] LNet: Removed LNI 192.168.206.54@tcp [ 1916.027447] Key type .llcrypt unregistered [ 1916.031424] Key type ._llcrypt unregistered [ 1921.743835] Key type ._llcrypt registered [ 1921.748772] Key type .llcrypt registered [ 1921.907751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.923203] alg: No test for adler32 (adler32-zlib) [ 1922.801562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1922.821789] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1922.847472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1928.449882] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:18:43 (1787858323) [ 1929.682759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1931.213221] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:18:46 (1787858326) [ 1932.553305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1934.142044] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:18:49 (1787858329) [ 1934.209624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.235444] LNet: 112966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1934.241134] LNetError: 112966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.257262] LNet: Removed LNI 192.168.206.54@tcp [ 1934.279160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.347397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1934.367472] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1935.124630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1935.161766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1941.226336] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:18:56 (1787858336) [ 1941.349730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.373830] LNet: 113679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.382916] LNetError: 113679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.404232] LNet: Removed LNI 192.168.206.54@tcp [ 1941.436063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.483239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1941.515664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1942.227293] Key type .llcrypt unregistered [ 1942.230225] Key type ._llcrypt unregistered [ 1952.901115] Key type ._llcrypt registered [ 1952.902784] Key type .llcrypt registered [ 1954.078961] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:19:09 (1787858349) [ 1954.459535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.477547] alg: No test for adler32 (adler32-zlib) [ 1955.416687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.466899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1955.487598] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1956.179718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1956.232896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1961.935650] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:19:17 (1787858357) [ 1962.135615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.151441] LNet: 115241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.160575] LNetError: 115241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.179265] LNet: Removed LNI 192.168.206.54@tcp [ 1962.229878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.279540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1962.302756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1967.688798] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:19:22 (1787858362) [ 1967.785465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.819454] LNet: 115816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.853397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.906242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1968.280528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.318125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1968.401255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1968.835785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.856337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1968.953957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.978915] LNet: 115854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.010516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.090536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1969.593867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1969.648253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1969.724550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1970.175728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1970.219703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1975.859436] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:19:30 (1787858370) [ 1975.950854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.992597] LNet: 116448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.013055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.106567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.128483] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1977.003207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1977.019089] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1977.790638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1977.839594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1978.039985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.073987] LNetError: 116733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.103361] LNet: Removed LNI 192.168.206.54@tcp [ 1979.105102] LNetError: 116733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.123804] LNet: Removed LNI 192.168.206.54@tcp2 [ 1979.154647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.232569] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1979.240982] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1979.247413] LNet: 116739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1979.741146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1979.931976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.959141] LNet: Removed LNI 192.168.206.54@tcp [ 1980.966893] LNet: Removed LNI 192.168.206.54@tcp2 [ 1980.993730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.047189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.076493] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1981.753943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1981.770134] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1982.586810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1982.636032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1982.688618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1982.916712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.935372] LNet: 117061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.939847] LNet: 117061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1982.944091] LNetError: 117061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.948098] LNetError: 117061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1982.953035] LNet: Removed LNI 192.168.206.54@tcp [ 1983.973788] LNet: Removed LNI 192.168.206.54@tcp2 [ 1984.018615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.141070] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1984.147694] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1984.156656] LNet: 117067:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1984.870203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1985.119087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.169882] LNet: Removed LNI 192.168.206.54@tcp [ 1986.229313] LNet: Removed LNI 192.168.206.54@tcp2 [ 1986.276816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.347511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1986.413066] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1987.167124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1987.186334] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1988.062822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1988.117498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1988.303433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.327974] LNetError: 117384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.333747] LNetError: 117384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1988.340984] LNet: Removed LNI 192.168.206.54@tcp [ 1988.360823] LNet: Removed LNI 192.168.206.54@tcp2 [ 1988.376083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.444538] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1988.455192] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1988.463108] LNet: 117390:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1989.045969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1989.351223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.431537] LNet: Removed LNI 192.168.206.54@tcp [ 1990.447297] LNet: Removed LNI 192.168.206.54@tcp2 [ 1990.487919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.552818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.596306] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1991.277638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1991.295711] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1992.130332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1992.182413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1992.283342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1992.499954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.520450] LNet: 117712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.523429] LNet: 117712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1993.568118] LNet: Removed LNI 192.168.206.54@tcp [ 1994.600289] LNet: Removed LNI 192.168.206.54@tcp2 [ 1994.619976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.672309] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1994.678896] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1994.682924] LNet: 117718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.151951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1995.304760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.324958] LNet: Removed LNI 192.168.206.54@tcp [ 1996.389730] LNet: Removed LNI 192.168.206.54@tcp2 [ 1996.415404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.471958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1996.501081] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1997.314967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1997.336204] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1998.242469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1998.267941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1998.317929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1998.686860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.702784] LNetError: 118052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.709536] LNetError: 118052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1998.715453] LNet: Removed LNI 192.168.206.54@tcp [ 1998.724702] LNet: Removed LNI 192.168.206.54@tcp2 [ 1998.748965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.804745] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 1998.809829] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 1998.814369] LNet: 118058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1999.544978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 1999.905281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.991318] LNet: Removed LNI 192.168.206.54@tcp [ 2001.009394] LNet: Removed LNI 192.168.206.54@tcp2 [ 2001.042898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.125782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.157129] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2001.746818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2001.769099] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2002.483433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2002.526446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2002.562120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2002.622918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2002.931426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.000950] LNet: Removed LNI 192.168.206.54@tcp [ 2005.030203] LNet: Removed LNI 192.168.206.54@tcp2 [ 2005.063381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.127192] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2005.131116] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2005.651709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2011.980790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2013.449101] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:20:08 (1787858408) [ 2013.715901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.754180] LNet: 119178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.762272] LNet: 119178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2013.777464] LNet: Removed LNI 192.168.206.54@tcp [ 2014.843682] LNet: Removed LNI 192.168.206.54@tcp2 [ 2014.879164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.244678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2015.274311] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2015.912795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2015.935653] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2016.592985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.617257] LNetError: 119452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2016.622152] LNetError: 119452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2016.627208] LNet: Removed LNI 10.1.2.3@tcp [ 2016.635831] LNet: Removed LNI 192.168.206.54@tcp [ 2016.653539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.694249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2016.709282] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2016.712816] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2017.147795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2017.192790] LNet: Removed LNI 192.168.206.54@tcp [ 2017.225374] LNet: Removed LNI 10.1.2.3@tcp [ 2022.642520] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:20:17 (1787858417) [ 2022.735606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.777746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.826676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.843812] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2023.515864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2023.555319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.611234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.673640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.722493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.774293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.833843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.900636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.932827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2023.994950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2024.042865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.54@tcp [ 2024.223820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2030.627679] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:20:25 (1787858425) [ 2030.792695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.910172] LNet: Removed LNI 192.168.206.54@tcp [ 2031.945274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.041205] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2032.130104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2032.133443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2032.136536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2034.530829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2034.560180] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2035.253146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2035.278323] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2037.132739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2037.179530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2037.246248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.54@tcp -m PUT -e local_timeout -r 1 [ 2037.310869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2037.351823] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2037.355252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2037.637386] LNetError: 120850:0:(lib-move.c:4801:lnet_parse()) 192.168.206.54@tcp, src 192.168.206.54@tcp: Dropping PUT (error -108 looking up sender) [ 2038.688066] LNet: Removed LNI 10.1.2.3@tcp [ 2039.720806] LNet: Removed LNI 192.168.206.54@tcp [ 2040.247647] Key type .llcrypt unregistered [ 2040.258141] Key type ._llcrypt unregistered [ 2052.973231] Key type ._llcrypt registered [ 2052.975845] Key type .llcrypt registered [ 2054.945954] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:20:49 (1787858449) [ 2055.574342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.596719] alg: No test for adler32 (adler32-zlib) [ 2056.644841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.176807] LNet: 122168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.303529] Key type .llcrypt unregistered [ 2058.309087] Key type ._llcrypt unregistered [ 2063.834575] Key type ._llcrypt registered [ 2063.844527] Key type .llcrypt registered [ 2064.143927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2064.160046] alg: No test for adler32 (adler32-zlib) [ 2065.112804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2065.158309] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2072.703399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 2072.870513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2073.348474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.154@tcp [ 2073.376302] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 2073.561179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2073.598633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2073.636068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2073.663977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2073.701469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2073.735464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2073.772424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2073.803422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2073.821965] LNet: 122621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2073.846883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2073.886829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2073.909944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2073.935990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2073.995042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2074.063091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2074.563432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2074.788180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2074.813216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2074.834573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2074.857715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2074.885956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2074.910515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2074.930897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2074.955460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2074.968572] LNet: 122738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2074.980708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2074.999575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2075.026061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2075.048858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2075.075703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2075.105588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2075.521183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.154@tcp [ 2075.671873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2075.688444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2075.713381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2075.743217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2075.761512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2075.787617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2075.817106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2075.839319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2075.863788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2075.885429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2075.908526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2075.934472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2075.961897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2075.980953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2076.368758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2076.555490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2076.576720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2076.596589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2076.613549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2076.634745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2076.655309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2076.684065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2076.726282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2076.746315] LNet: 122973:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2076.751392] LNet: 122973:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2076.766216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2076.801175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2076.821848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2076.854104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2076.887919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2076.916532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2077.187818] LNet: 123023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.194233] LNetError: 123023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.207984] LNet: Removed LNI 192.168.206.54@tcp [ 2077.577230] Key type .llcrypt unregistered [ 2077.581328] Key type ._llcrypt unregistered [ 2093.901503] Key type ._llcrypt registered [ 2093.916837] Key type .llcrypt registered [ 2095.458871] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:21:30 (1787858490) [ 2096.317336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.346463] alg: No test for adler32 (adler32-zlib) [ 2097.371917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.964482] LNet: 123909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.607979] Key type .llcrypt unregistered [ 2098.610522] Key type ._llcrypt unregistered [ 2105.482521] Key type ._llcrypt registered [ 2105.487107] Key type .llcrypt registered [ 2105.770462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2105.807542] alg: No test for adler32 (adler32-zlib) [ 2106.806963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2106.831166] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2112.975953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.154@tcp [ 2120.322403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2120.361370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2120.402701] LNet: Added LNI 192.168.206.54@tcp1 [8/256/0/180] [ 2121.406485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2121.964754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.154@tcp [ 2122.211761] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.154@tcp [ 2122.237116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2122.288332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2122.343898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2122.372727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2122.423240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2122.472404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2122.508766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2122.567204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2122.602738] LNet: 124564:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2122.622926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2122.668639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2122.725309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2122.781647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2122.845936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2122.907168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2123.492887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2123.724659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2123.751595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2123.791928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2123.848955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2123.895880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2123.960084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2123.989457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2124.008819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2124.027153] LNet: 124681:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2124.050723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2124.097046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2124.145773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2124.180915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2124.228589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2124.270142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2124.863754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.154@tcp [ 2125.171671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2125.207496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2125.271388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2125.330769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2125.366939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2125.405028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2125.463078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2125.522257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2125.563741] LNet: 124798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2125.600923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2125.651944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2125.688299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2125.729973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2125.769661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2125.799329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2126.423621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2126.624808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2126.652416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2126.679685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2126.704888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2126.731170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2126.756363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2126.783405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2126.811633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2126.861972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2126.920185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2126.955092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2126.983216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2127.020135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2127.060357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2131.664750] LNet: 125000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.678747] LNetError: 125000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.692925] LNet: Removed LNI 192.168.206.54@tcp [ 2131.707524] LNet: Removed LNI 192.168.206.54@tcp1 [ 2132.139725] Key type .llcrypt unregistered [ 2132.141714] Key type ._llcrypt unregistered [ 2146.919208] Key type ._llcrypt registered [ 2146.923088] Key type .llcrypt registered [ 2148.109792] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:22:23 (1787858543) [ 2148.488929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.507542] alg: No test for adler32 (adler32-zlib) [ 2149.412660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.486755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.506762] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2150.129970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2150.156763] LNet: Added LNI 192.168.206.54@tcp2 [8/256/0/180] [ 2150.906676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.54@tcp [ 2151.007179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2152.031918] LNet: Removed LNI 192.168.206.54@tcp [ 2157.867574] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:22:32 (1787858552) [ 2159.252754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2160.869555] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:22:35 (1787858555) [ 2167.799921] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:22:42 (1787858562) [ 2175.339255] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:22:50 (1787858570) [ 2181.874575] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:22:56 (1787858576) [ 2189.405202] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:23:04 (1787858584) [ 2197.396423] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:23:12 (1787858592) [ 2205.393872] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:23:20 (1787858600) [ 2213.382209] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:23:28 (1787858608) [ 2220.352767] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:23:35 (1787858615) [ 2226.513877] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:23:41 (1787858621) [ 2232.629349] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:23:47 (1787858627) [ 2238.557611] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:23:53 (1787858633) [ 2244.779523] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:23:59 (1787858639) [ 2251.286330] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:24:06 (1787858646) [ 2257.657496] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:24:12 (1787858652) [ 2263.098984] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:24:18 (1787858658) [ 2269.090535] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:24:24 (1787858664) [ 2276.249886] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:24:31 (1787858671) [ 2276.347514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.372991] LNet: 136686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.380328] LNetError: 136686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.398762] LNet: Removed LNI 192.168.206.54@tcp2 [ 2276.422973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.546477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2276.577752] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2277.592821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2277.615418] LNet: 136834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.623720] LNetError: 136834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.640812] LNet: Removed LNI 192.168.206.54@tcp [ 2277.683895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.776383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2277.808066] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2284.227254] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:24:39 (1787858679) [ 2290.936690] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:24:46 (1787858686) [ 2297.184711] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:24:52 (1787858692) [ 2304.141251] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:24:59 (1787858699) [ 2311.021546] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:25:05 (1787858705) [ 2319.000401] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:25:14 (1787858714) [ 2326.454544] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:25:21 (1787858721) [ 2333.187624] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:25:28 (1787858728) [ 2340.040736] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:25:34 (1787858734) [ 2347.815176] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:25:42 (1787858742) [ 2354.419684] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:25:49 (1787858749) [ 2361.122434] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:25:56 (1787858756) [ 2366.806885] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2138 sec ========== 15:26:01 (1787858761) [ 2367.113676] LNet: 144299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.118945] LNetError: 144299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.131581] LNet: Removed LNI 192.168.206.54@tcp [ 2367.728499] Key type .llcrypt unregistered [ 2367.731069] Key type ._llcrypt unregistered [ 2379.284112] Key type ._llcrypt registered [ 2379.288483] Key type .llcrypt registered [ 2379.591275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2379.603336] alg: No test for adler32 (adler32-zlib) [ 2380.620599] Lustre: Lustre: Build Version: 2.17.57_81_g80478ec [ 2380.890949] LNet: Added LNI 192.168.206.54@tcp [8/256/0/180] [ 2382.567278] Key type lgssc registered [ 2383.500402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.046887] Lustre: Mounted lustre-client - version 2.17.57_81_g80478ec [ 2453.435833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2461.082682] Lustre: Unmounted lustre-client [ 2486.536610] Key type lgssc unregistered [ 2486.778062] LNet: 147256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2486.790630] LNetError: 147256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2486.805207] LNet: Removed LNI 192.168.206.54@tcp [ 2487.338155] Key type .llcrypt unregistered [ 2487.341102] Key type ._llcrypt unregistered