[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454720342 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003038] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008273] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011122] LSM: Security Framework initializing [ 0.012040] Yama: becoming mindful. [ 0.013026] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.021817] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025685] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] *** VALIDATE tmpfs *** [ 0.030237] *** VALIDATE proc *** [ 0.031237] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037007] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041069] debug: unmapping init [mem 0xffffffff9a659000-0xffffffff9a660fff] [ 0.044127] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047032] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053401] rcu: Hierarchical SRCU implementation. [ 0.055701] smp: Bringing up secondary CPUs ... [ 0.056644] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.060714] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.168066] node 0 deferred pages initialised in 100ms [ 0.170559] devtmpfs: initialized [ 0.171185] x86/mm: Memory block size: 128MB [ 0.173602] gcov: version magic: 0x41383552 [ 0.175234] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.176078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177235] pinctrl core: initialized pinctrl subsystem [ 0.178139] [ 0.178602] ************************************************************* [ 0.179011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180014] ** ** [ 0.181011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.182074] ** ** [ 0.183012] ** This means that this kernel is built to expose internal ** [ 0.184010] ** IOMMU data structures, which may compromise security on ** [ 0.185013] ** your system. ** [ 0.186011] ** ** [ 0.187012] ** If you see this message and you are not debugging the ** [ 0.188016] ** kernel, report this immediately to your vendor! ** [ 0.189015] ** ** [ 0.190013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191016] ************************************************************* [ 0.192736] NET: Registered protocol family 16 [ 0.193504] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196432] cpuidle: using governor menu [ 0.199034] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201922] PCI: Using configuration type 1 for base access [ 0.205149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223178] cryptd: max_cpu_qlen set to 1000 [ 0.225366] ACPI: Added _OSI(Module Device) [ 0.227023] ACPI: Added _OSI(Processor Device) [ 0.228019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235444] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243542] ACPI: Interpreter enabled [ 0.244048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245011] ACPI: Using IOAPIC for interrupt routing [ 0.247119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249556] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259409] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270331] acpiphp: Slot [2] registered [ 0.271084] acpiphp: Slot [5] registered [ 0.272087] acpiphp: Slot [6] registered [ 0.273081] acpiphp: Slot [3] registered [ 0.274072] acpiphp: Slot [4] registered [ 0.275074] acpiphp: Slot [7] registered [ 0.277088] acpiphp: Slot [8] registered [ 0.278082] acpiphp: Slot [9] registered [ 0.279089] acpiphp: Slot [10] registered [ 0.280140] acpiphp: Slot [11] registered [ 0.282060] acpiphp: Slot [12] registered [ 0.283090] acpiphp: Slot [13] registered [ 0.284084] acpiphp: Slot [14] registered [ 0.285076] acpiphp: Slot [15] registered [ 0.286058] acpiphp: Slot [16] registered [ 0.287087] acpiphp: Slot [17] registered [ 0.289081] acpiphp: Slot [18] registered [ 0.290090] acpiphp: Slot [19] registered [ 0.291104] acpiphp: Slot [20] registered [ 0.292084] acpiphp: Slot [21] registered [ 0.293082] acpiphp: Slot [22] registered [ 0.294091] acpiphp: Slot [23] registered [ 0.295089] acpiphp: Slot [24] registered [ 0.297115] acpiphp: Slot [25] registered [ 0.298096] acpiphp: Slot [26] registered [ 0.299106] acpiphp: Slot [27] registered [ 0.301121] acpiphp: Slot [28] registered [ 0.302091] acpiphp: Slot [29] registered [ 0.303075] acpiphp: Slot [30] registered [ 0.304116] acpiphp: Slot [31] registered [ 0.306090] PCI host bridge to bus 0000:00 [ 0.307017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.310023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.312020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.315021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.319025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.321175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324192] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327286] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335022] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.339102] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.347016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348569] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.355041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356787] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.370015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.374014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.377377] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.391015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401936] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.410041] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.415016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.436000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.445778] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.447374] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.449371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.452421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.455269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.460155] iommu: Default domain type: Passthrough [ 0.463312] SCSI subsystem initialized [ 0.464096] ACPI: bus type USB registered [ 0.466139] usbcore: registered new interface driver usbfs [ 0.468103] usbcore: registered new interface driver hub [ 0.469075] usbcore: registered new device driver usb [ 0.471189] pps_core: LinuxPPS API ver. 1 registered [ 0.473015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.477082] PTP clock support registered [ 0.479110] EDAC MC: Ver: 3.0.0 [ 0.480475] PCI: Using ACPI for IRQ routing [ 0.482956] NetLabel: Initializing [ 0.484023] NetLabel: domain hash size = 128 [ 0.486014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.488084] NetLabel: unlabeled traffic allowed by default [ 0.492109] vgaarb: loaded [ 0.494336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.496023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.504000] clocksource: Switched to clocksource kvm-clock [ 0.611467] VFS: Disk quotas dquot_6.6.0 [ 0.613881] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.617913] *** VALIDATE ramfs *** [ 0.620257] *** VALIDATE hugetlbfs *** [ 0.622605] pnp: PnP ACPI init [ 0.626425] pnp: PnP ACPI: found 6 devices [ 0.646629] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.651209] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.653753] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.657326] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.660192] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.663147] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.665412] NET: Registered protocol family 2 [ 0.668061] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.673584] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.678022] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.683348] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.687127] TCP: Hash tables configured (established 65536 bind 65536) [ 0.690059] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.693509] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.696739] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.700147] NET: Registered protocol family 1 [ 0.702650] RPC: Registered named UNIX socket transport module. [ 0.705124] RPC: Registered udp transport module. [ 0.706407] RPC: Registered tcp transport module. [ 0.707709] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.709419] NET: Registered protocol family 44 [ 0.710724] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.712296] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.713928] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.716530] PCI: CLS 0 bytes, default 64 [ 0.718409] Unpacking initramfs... [ 2.099397] debug: unmapping init [mem 0xffff8fb77cc64000-0xffff8fb77ffcffff] [ 2.103315] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.105186] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.107411] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.651247] Initialise system trusted keyrings [ 2.654324] Key type blacklist registered [ 2.658242] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.668709] zbud: loaded [ 2.672178] *** VALIDATE nfs *** [ 2.673140] *** VALIDATE nfs4 *** [ 2.676444] pstore: using deflate compression [ 2.681071] Platform Keyring initialized [ 2.809799] NET: Registered protocol family 38 [ 2.811743] Key type asymmetric registered [ 2.813368] Asymmetric key parser 'x509' registered [ 2.815209] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.818103] io scheduler mq-deadline registered [ 2.820076] io scheduler kyber registered [ 2.821841] io scheduler bfq registered [ 2.823890] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.827624] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.830857] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.834342] ACPI: Power Button [PWRF] [ 2.840068] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.847272] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.860783] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.889899] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.921853] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.927395] Non-volatile memory driver v1.3 [ 2.929014] Linux agpgart interface v0.103 [ 2.965168] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.969964] vda: detected capacity change from 0 to 74698752 [ 2.986738] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.989728] vdb: detected capacity change from 0 to 1073741824 [ 2.996895] libphy: Fixed MDIO Bus: probed [ 3.004204] usbcore: registered new interface driver usbserial_generic [ 3.007171] usbserial: USB Serial support registered for generic [ 3.009784] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.014217] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.016238] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.019214] mousedev: PS/2 mouse device common for all mice [ 3.023242] rtc_cmos 00:05: RTC can wake from S4 [ 3.025476] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.028135] rtc_cmos 00:05: registered as rtc0 [ 3.034981] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.037219] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.042528] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.043530] intel_pstate: CPU model not supported [ 3.049097] hid: raw HID events driver (C) Jiri Kosina [ 3.051621] usbcore: registered new interface driver usbhid [ 3.054027] usbhid: USB HID core driver [ 3.055196] drop_monitor: Initializing network drop monitor service [ 3.057545] Initializing XFRM netlink socket [ 3.059537] NET: Registered protocol family 10 [ 3.061672] Segment Routing with IPv6 [ 3.062875] NET: Registered protocol family 17 [ 3.064551] mpls_gso: MPLS GSO support [ 3.070950] RAS: Correctable Errors collector initialized. [ 3.073484] AVX version of gcm_enc/dec engaged. [ 3.075103] AES CTR mode by8 optimization enabled [ 3.153126] sched_clock: Marking stable (3153101922, 0)->(4118586290, -965484368) [ 3.156569] registered taskstats version 1 [ 3.158781] Loading compiled-in X.509 certificates [ 3.160560] zswap: loaded using pool lzo/zbud [ 3.186401] Key type big_key registered [ 3.198752] Key type encrypted registered [ 3.200505] ima: No TPM chip found, activating TPM-bypass! [ 3.202228] ima: Allocated hash algorithm: sha1 [ 3.204038] ima: No architecture policies found [ 3.205403] evm: Initialising EVM extended attributes: [ 3.207254] evm: security.selinux [ 3.208410] evm: security.ima [ 3.209483] evm: security.capability [ 3.210757] evm: HMAC attrs: 0x1 [ 3.212905] rtc_cmos 00:05: setting system clock to 2026-08-14 06:30:14 UTC (1786689014) [ 3.219262] debug: unmapping init [mem 0xffffffff9b603000-0xffffffff9b7fffff] [ 3.222896] debug: unmapping init [mem 0xffffffff9a382000-0xffffffff9a658fff] [ 3.235142] Write protecting the kernel read-only data: 28672k [ 3.238990] debug: unmapping init [mem 0xffffffff98a03000-0xffffffff98bfffff] [ 3.241832] debug: unmapping init [mem 0xffffffff99314000-0xffffffff993fffff] [ 3.290962] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.300113] systemd[1]: Detected virtualization kvm. [ 3.301413] systemd[1]: Detected architecture x86-64. [ 3.302782] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.326939] systemd[1]: No hostname configured. [ 3.328368] systemd[1]: Set hostname to . [ 3.330436] random: systemd: uninitialized urandom read (16 bytes read) [ 3.332886] systemd[1]: Initializing machine ID from random generator. [ 3.368664] random: ln: uninitialized urandom read (6 bytes read) [ 3.459040] random: systemd: uninitialized urandom read (16 bytes read) [ 3.462275] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.466590] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.471737] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.145930] device-mapper: uevent: version 1.0.3 [ 4.148130] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 4.488914] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.927507] virtio_net virtio0 ens2: renamed from eth0 [ 4.989092] scsi host0: ata_piix [ 4.993910] scsi host1: ata_piix [ 4.995636] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.999982] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.473458] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.722909] random: crng init done [ 9.725164] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.624944] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.421252] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.425251] SELinux: Disabled at runtime. [ 15.543848] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.558184] systemd[1]: Detected virtualization kvm. [ 15.562541] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.019061] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.034496] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.058457] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.065494] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.077984] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.106382] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.133880] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK [ 17.682391] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.979075] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.227981] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.365909] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.956613] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.118654] EDAC sbridge: Ver: 1.1.2 [ 24.761654] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit)[ 25.550948] NFS: Registering the id_resolver key type [ 25.566053] Key type id_resolver registered [ 25.570381] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg647-client login: [ 87.153664] libcfs: loading out-of-tree module taints kernel. [ 87.404315] Key type ._llcrypt registered [ 87.412401] Key type .llcrypt registered [ 87.803414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 87.821485] alg: No test for adler32 (adler32-zlib) [ 89.454528] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 90.934196] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 92.879218] Key type lgssc registered [ 94.813515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 215.847875] Lustre: Mounted lustre-client - version 2.17.57_1_g9a8e296 [ 220.884554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.321090] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing check_logdir /tmp/testlogs/ [ 239.305809] Lustre: DEBUG MARKER: oleg647-client.virtnet: executing yml_node [ 241.632124] Lustre: lustre-OST0000-osc-ffff8fb7c700f000: disconnect after 23s idle [ 243.144409] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 245.044110] hrtimer: interrupt took 2121648 ns [ 245.278216] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 247.918080] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 249.339164] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 02:34:19 EDT 2026 [ 264.398773] Lustre: Unmounted lustre-client [ 279.814868] Key type lgssc unregistered [ 280.063664] LNet: 8414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.067900] LNetError: 8414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 280.086072] LNet: Removed LNI 192.168.206.47@tcp [ 280.835147] Key type .llcrypt unregistered [ 280.837393] Key type ._llcrypt unregistered [ 303.181976] Key type ._llcrypt registered [ 303.184660] Key type .llcrypt registered [ 303.360257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 303.370759] alg: No test for adler32 (adler32-zlib) [ 304.271253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 304.334254] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 304.355950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 307.885595] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 308.113462] LNet: 9301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.121444] LNetError: 9301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 308.151312] LNet: Removed LNI 192.168.206.47@tcp [ 308.829388] Key type .llcrypt unregistered [ 308.834916] Key type ._llcrypt unregistered [ 315.321676] Key type ._llcrypt registered [ 315.325275] Key type .llcrypt registered [ 316.636883] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:35:26 (1786689326) [ 316.869183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 316.878960] alg: No test for adler32 (adler32-zlib) [ 317.816611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.940469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.958753] LNet: 9716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.987467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.057646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 318.081644] LNet: 9723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 325.678337] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:35:35 (1786689335) [ 325.888127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.914165] LNet: 10300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.946760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.018743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 334.309836] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:35:44 (1786689344) [ 334.497101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.536749] LNet: 10885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.566424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.677570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 343.379277] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:35:53 (1786689353) [ 343.887801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.966876] LNet: 11471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.001352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.146376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 351.055135] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:36:00 (1786689360) [ 351.179354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.206118] LNet: 12055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.254629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.328657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 351.833460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 358.558649] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:36:08 (1786689368) [ 358.707388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.776327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.892472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 359.498973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 359.859942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 365.483779] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:36:15 (1786689375) [ 365.559374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.594246] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.605195] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 365.631320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.717180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 371.102651] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:36:21 (1786689381) [ 371.236406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.311475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.359626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 371.390611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 371.650637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 371.681287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 372.135841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 372.172635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 372.491037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 372.550956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 372.771412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 372.805622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.071775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 373.113675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 373.423633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 373.446577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 373.792445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 373.847894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.246100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 374.280521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.716458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 374.750308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.133472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 375.165516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.416527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 375.454634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 375.690435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 375.722758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 381.315601] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:36:31 (1786689391) [ 381.636821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.655799] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.659218] LNet: 14511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 381.673258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.753669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 381.787945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 387.772938] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:36:37 (1786689397) [ 388.162328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.194328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.296642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 388.344632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 395.679990] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:36:45 (1786689405) [ 395.777240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.839855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.958187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 396.011335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 402.449121] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:36:52 (1786689412) [ 402.593473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.656152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.746203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 402.782051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 409.040313] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:36:59 (1786689419) [ 409.116724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.163915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.276817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 409.309374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 415.113815] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:37:04 (1786689424) [ 415.187752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.199771] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.208148] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 415.262194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.375935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 415.426647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 421.090748] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:37:11 (1786689431) [ 421.262966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.330092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.466400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 421.786206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.911653] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:37:17 (1786689437) [ 428.033679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.098591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.335161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.511295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.567218] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:37:24 (1786689444) [ 434.669039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.709097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.866770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.056544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.145919] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:37:31 (1786689451) [ 441.265396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.321332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.868901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.180917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.524790] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:37:38 (1786689458) [ 448.606935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.676862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.763770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.836536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.877640] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:37:44 (1786689464) [ 455.102642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.140363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.195111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.245357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.104792] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:37:51 (1786689471) [ 462.265297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.322627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.425566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 468.898807] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:37:58 (1786689478) [ 469.114516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.153329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.264753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 475.425608] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:38:05 (1786689485) [ 475.517725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.568511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.635361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 475.996355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 481.515070] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:38:11 (1786689491) [ 481.587323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.622970] LNet: 23462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.635704] LNet: 23462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 481.663843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.779792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 481.829460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 487.472358] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:38:17 (1786689497) [ 487.568986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.613140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.717370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 487.744910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 494.177819] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:38:24 (1786689504) [ 494.428963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.489471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.606595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 494.641595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 500.136432] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:38:30 (1786689510) [ 500.344477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.395943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.499578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 500.530429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 507.080547] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:38:37 (1786689517) [ 507.211329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.270205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.353308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 507.436526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 507.458423] LNetError: 25826:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 507.490508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 507.534559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 513.530847] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:38:43 (1786689523) [ 513.634077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.668920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.775658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zq0bsTOMk6 [ 513.868407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zq0bsTOMk6 [ 513.924629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zq0bsTOMk6 [ 519.527633] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:38:49 (1786689529) [ 519.748907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.781732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.852643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 519.913395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 519.999436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 525.570414] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:38:55 (1786689535) [ 525.735659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.810686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.864823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.893533] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 527.362928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 528.597247] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 530.600905] Key type lgssc registered [ 531.663103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.606691] LNetError: 27587:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.47@tcp99 has no available nets [ 536.890237] Key type lgssc unregistered [ 537.238530] LNetError: 28696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.263660] LNet: Removed LNI 192.168.206.47@tcp [ 538.033646] Key type .llcrypt unregistered [ 538.036214] Key type ._llcrypt unregistered [ 543.593126] Key type ._llcrypt registered [ 543.595129] Key type .llcrypt registered [ 544.807704] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:39:14 (1786689554) [ 545.154305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 545.170370] alg: No test for adler32 (adler32-zlib) [ 546.082466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.182959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 548.325382] LNet: Message forwarding will be enabled in 2s [ 551.391136] LNet: Message forwarding enabled [ 551.833252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.857746] LNet: Message forwarding will stop in 2s to 6s [ 556.359386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.984793] LNet: Message forwarding disabled [ 558.525556] LNet: Message forwarding will be enabled in 2s [ 562.143262] LNet: Message forwarding enabled [ 564.335780] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:39:34 (1786689574) [ 564.453952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.491556] LNet: 30100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.617721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.692321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 570.750904] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:39:40 (1786689580) [ 570.845680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.865668] LNet: 30679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.887995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.957294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 570.985383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 571.012634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 571.048593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 571.082602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 571.104665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 571.130429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 571.155769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 571.188334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 571.223984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 571.261431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 571.293556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 571.319345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 571.351290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 571.373904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 571.402737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 571.440926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 571.469797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 571.496610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 571.511969] LNetError: 30729:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 576.604088] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:39:46 (1786689586) [ 576.682952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.719879] LNet: 31295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.753914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.811874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 582.737370] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:39:52 (1786689592) [ 584.306409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg647-server [ 585.982332] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:39:56 (1786689596) [ 586.130278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.158398] LNet: 32230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.196042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.255855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.283330] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 586.934481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[48-52]@tcp [ 586.958460] LNet: 32363:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 586.992660] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.49@tcp is being used as a gateway but routing feature is not turned on [ 587.013349] LNetError: 32234:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.49@tcp has gone from up to down [ 588.015836] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.49@tcp is being used as a gateway but routing feature is not turned on [ 588.024541] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 589.026355] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.48@tcp is being used as a gateway but routing feature is not turned on [ 589.039108] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 591.079801] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.50@tcp is being used as a gateway but routing feature is not turned on [ 591.091284] LNetError: 32234:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 591.899871] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:40:02 (1786689602) [ 593.193330] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg647-server [ 594.357643] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:40:04 (1786689604) [ 594.431479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.445249] LNet: 33290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.449767] LNetError: 33290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.467345] LNet: Removed LNI 192.168.206.47@tcp [ 594.484889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.556134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.578489] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 595.154958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[48-53/2]@tcp [ 595.174765] LNet: 33415:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 595.183582] LNet: 33415:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 595.205961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[48-53/2]@tcp [ 595.208104] LNetError: 33301:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.48@tcp is being used as a gateway but routing feature is not turned on [ 595.225444] LNetError: 33301:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 595.230601] LNetError: 33301:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.48@tcp has gone from up to down [ 595.236352] LNetError: 33301:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 600.385975] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:40:10 (1786689610) [ 600.487890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.542219] LNetError: 33999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 600.567344] LNet: Removed LNI 192.168.206.47@tcp [ 600.598212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.712248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 600.775794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.834930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.910318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.980693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.019820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.075619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 601.149687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.187159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.246915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.342396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.391399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.447625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 601.525445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.559055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.619843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.716818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.762612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.834808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 601.935483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.978766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.037442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.122594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.192874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.240805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 602.311615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.337827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.400933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.472893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.485831] LNet: 34142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.489743] LNet: 34142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 602.505466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.551572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 602.630367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.684514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.743455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 608.330348] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:40:18 (1786689618) [ 610.149252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg647-server [ 611.441222] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:40:21 (1786689621) [ 613.107186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg647-server [ 614.401301] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:40:24 (1786689624) [ 615.049539] Key type .llcrypt unregistered [ 615.054852] Key type ._llcrypt unregistered [ 619.559365] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 619.600019] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 619.604503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 619.607690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 622.089603] Key type ._llcrypt registered [ 622.091571] Key type .llcrypt registered [ 622.293909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 622.306552] alg: No test for adler32 (adler32-zlib) [ 623.231187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.296570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.333529] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 624.000347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 624.019322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 625.585585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 625.601836] LNet: Removed LNI 10.1.2.3@tcp [ 625.752679] LNet: 36127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.760130] LNetError: 36127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.791247] LNet: Removed LNI 192.168.206.47@tcp [ 626.464177] Key type .llcrypt unregistered [ 626.465821] Key type ._llcrypt unregistered [ 637.763074] Key type ._llcrypt registered [ 637.764946] Key type .llcrypt registered [ 639.151810] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:40:48 (1786689648) [ 639.646460] Key type .llcrypt unregistered [ 639.651271] Key type ._llcrypt unregistered [ 646.541950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 646.598412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 646.600902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 646.604204] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 648.989862] Key type ._llcrypt registered [ 648.993417] Key type .llcrypt registered [ 649.134258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 649.146667] alg: No test for adler32 (adler32-zlib) [ 650.001751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.043889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 650.064732] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 650.670805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 650.693785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.674193] LNet: 37628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.678508] LNetError: 37628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 652.688971] LNet: Removed LNI 10.1.2.3@tcp [ 652.704965] LNet: Removed LNI 192.168.206.47@tcp [ 653.402401] Key type .llcrypt unregistered [ 653.406621] Key type ._llcrypt unregistered [ 663.356757] Key type ._llcrypt registered [ 663.359705] Key type .llcrypt registered [ 664.724591] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:41:14 (1786689674) [ 665.246459] Key type .llcrypt unregistered [ 665.247964] Key type ._llcrypt unregistered [ 670.205852] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 670.327034] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 670.332779] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 670.336486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 672.797497] Key type ._llcrypt registered [ 672.799624] Key type .llcrypt registered [ 672.951803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 672.965772] alg: No test for adler32 (adler32-zlib) [ 673.868811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.923231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 673.952158] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 675.475595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 675.500849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 677.170289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 677.190951] LNet: Removed LNI 10.1.2.3@tcp [ 677.216559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 677.272820] LNet: Removed LNI 10.1.2.31@tcp [ 677.298423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.330075] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 678.033255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 678.055237] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 678.905387] LNet: 39559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.916125] LNetError: 39559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 678.926520] LNet: Removed LNI 10.1.2.31@tcp [ 678.943831] LNet: Removed LNI 10.1.2.3@tcp [ 679.513331] Key type .llcrypt unregistered [ 679.517475] Key type ._llcrypt unregistered [ 689.505146] Key type ._llcrypt registered [ 689.507769] Key type .llcrypt registered [ 690.595354] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:41:40 (1786689700) [ 690.926234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.940043] alg: No test for adler32 (adler32-zlib) [ 691.853247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.896406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.916956] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 692.517200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.529755] LNet: 40517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.533498] LNetError: 40517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.549324] LNet: Removed LNI 192.168.206.47@tcp [ 692.567633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.619509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.47@tcp [ 692.645161] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 693.325203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.47@tcp [ 693.349796] LNet: Removed LNI 192.168.206.47@tcp [ 693.397684] LNet: 40652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.015709] Key type .llcrypt unregistered [ 694.018879] Key type ._llcrypt unregistered [ 703.844366] Key type ._llcrypt registered [ 703.848227] Key type .llcrypt registered [ 705.035910] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:41:54 (1786689714) [ 706.578159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg647-server [ 707.938156] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:41:58 (1786689718) [ 708.330473] Key type .llcrypt unregistered [ 708.332155] Key type ._llcrypt unregistered [ 713.895718] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 714.119125] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 714.123092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 714.125657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 716.600302] Key type ._llcrypt registered [ 716.603630] Key type .llcrypt registered [ 716.757198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.777823] alg: No test for adler32 (adler32-zlib) [ 717.662609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.725489] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 718.516955] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 720.546795] LNet: 42497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.552678] LNetError: 42497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.558561] LNet: Removed LNI 192.168.206.47@tcp1 [ 720.569779] LNet: Removed LNI 10.1.2.3@tcp2 [ 721.306417] Key type .llcrypt unregistered [ 721.308728] Key type ._llcrypt unregistered [ 731.518770] Key type ._llcrypt registered [ 731.522595] Key type .llcrypt registered [ 732.725549] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:42:22 (1786689742) [ 733.190923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.206115] alg: No test for adler32 (adler32-zlib) [ 734.092763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.148068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.183551] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 734.950896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.962863] LNet: 43471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.969643] LNetError: 43471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.979755] LNet: Removed LNI 192.168.206.47@tcp [ 734.995319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.052357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 735.073645] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 741.726869] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:42:31 (1786689751) [ 742.078084] LNet: 44062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.087674] LNetError: 44062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.119193] LNet: Removed LNI 192.168.206.47@tcp [ 742.697742] Key type .llcrypt unregistered [ 742.699533] Key type ._llcrypt unregistered [ 747.863258] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 748.060554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 748.069400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 748.076275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.691031] Key type ._llcrypt registered [ 750.692906] Key type .llcrypt registered [ 750.931698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.951691] alg: No test for adler32 (adler32-zlib) [ 751.914526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.978465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.006868] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 752.994713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 753.011025] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 755.961035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.990286] LNet: 44805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.997425] LNetError: 44805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 756.009908] LNet: Removed LNI 10.1.2.3@tcp [ 756.021189] LNet: Removed LNI 192.168.206.47@tcp [ 756.035043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.146397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 756.210726] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 756.225773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 756.982351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.033269] LNet: 44832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.055830] LNetError: 44832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 757.069497] LNetError: 44832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 757.085528] LNet: Removed LNI 10.1.2.3@tcp [ 757.098719] LNet: Removed LNI 192.168.206.47@tcp [ 757.131594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.247580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 757.284432] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 757.287989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.778278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 757.848303] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 759.312854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.336809] LNet: 44994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.341259] LNetError: 44994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 759.348114] LNetError: 44994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 759.354900] LNet: Removed LNI 10.1.2.3@tcp [ 759.357229] LNet: Removed LNI 192.168.206.47@tcp [ 759.371790] LNet: Removed LNI 192.168.206.47@tcp2 [ 759.427117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.478194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 759.520398] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 759.529239] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.542822] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 770.325951] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:42:59 (1786689779) [ 770.454686] LNet: 45582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.477776] LNetError: 45582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 770.494380] LNetError: 45582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 770.509892] LNet: Removed LNI 10.1.2.3@tcp [ 770.512019] LNet: Removed LNI 192.168.206.47@tcp [ 770.540256] LNet: Removed LNI 192.168.206.47@tcp2 [ 771.693658] Key type .llcrypt unregistered [ 771.695058] Key type ._llcrypt unregistered [ 777.587703] Key type ._llcrypt registered [ 777.589070] Key type .llcrypt registered [ 777.757912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.779123] alg: No test for adler32 (adler32-zlib) [ 778.856385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.937942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.967397] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 780.231246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.248923] LNet: 45989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.257294] LNetError: 45989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.270475] LNet: Removed LNI 192.168.206.47@tcp [ 780.319311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.470294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 780.509541] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 781.081625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.100458] LNet: 46020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.104723] LNetError: 46020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.118565] LNet: Removed LNI 192.168.206.47@tcp [ 781.135979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.225888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.282058] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 781.827608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.863037] LNet: Removed LNI 192.168.206.47@tcp [ 781.884520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.943832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.969609] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 782.294275] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.340433] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.343423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.346203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.834957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.858704] LNet: 46127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.869187] LNet: 46127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 784.875137] LNetError: 46127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.880685] LNetError: 46127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.906412] LNet: Removed LNI 192.168.206.47@tcp [ 784.949620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.076040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 785.101279] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 793.560574] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:43:23 (1786689803) [ 793.703650] LNet: 46717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.712831] LNetError: 46717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.772274] LNet: Removed LNI 192.168.206.47@tcp [ 794.360684] Key type .llcrypt unregistered [ 794.364123] Key type ._llcrypt unregistered [ 800.636352] Key type ._llcrypt registered [ 800.639758] Key type .llcrypt registered [ 800.802122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.816184] alg: No test for adler32 (adler32-zlib) [ 801.774696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.851404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.894992] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 802.875963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.900396] LNet: 47128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.909983] LNetError: 47128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.930774] LNet: Removed LNI 192.168.206.47@tcp [ 802.954338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.008470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 803.040534] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 809.383052] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:43:39 (1786689819) [ 809.552519] LNet: 47720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.560422] LNetError: 47720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.578116] LNet: Removed LNI 192.168.206.47@tcp [ 810.264349] Key type .llcrypt unregistered [ 810.266148] Key type ._llcrypt unregistered [ 816.382151] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 816.497050] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 816.505500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 816.523536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 819.208125] Key type ._llcrypt registered [ 819.209832] Key type .llcrypt registered [ 819.407107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.417604] alg: No test for adler32 (adler32-zlib) [ 820.324674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.371777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.390076] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 821.145398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.172336] LNet: 48175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.177685] LNetError: 48175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.194146] LNet: Removed LNI 192.168.206.47@tcp [ 821.227635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.301793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 821.343626] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 827.473668] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:43:57 (1786689837) [ 827.636447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.650964] LNet: 48772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.654670] LNetError: 48772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.663567] LNet: Removed LNI 192.168.206.47@tcp [ 827.678936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.786959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 832.178802] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:44:02 (1786689842) [ 832.242448] LNet: 49345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.780668] Key type .llcrypt unregistered [ 832.785134] Key type ._llcrypt unregistered [ 837.463787] Key type ._llcrypt registered [ 837.467458] Key type .llcrypt registered [ 837.608745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.618230] alg: No test for adler32 (adler32-zlib) [ 838.511769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.558504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.574636] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 839.205109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.225865] LNet: 49755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.232894] LNetError: 49755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.251766] LNet: Removed LNI 192.168.206.47@tcp [ 839.264834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.307355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 839.322323] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 844.514929] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:44:14 (1786689854) [ 844.609201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.635564] LNet: 50343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.649420] LNetError: 50343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.667457] LNet: Removed LNI 192.168.206.47@tcp [ 844.691126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.759690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 849.875527] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:44:19 (1786689859) [ 851.185340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 852.959289] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:44:22 (1786689862) [ 854.385488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 855.655493] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:44:25 (1786689865) [ 856.945085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 858.489438] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:44:28 (1786689868) [ 859.860410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 861.216594] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:44:31 (1786689871) [ 862.756204] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 864.176990] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:44:34 (1786689874) [ 864.342298] LNet: 52400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.863742] Key type .llcrypt unregistered [ 864.866959] Key type ._llcrypt unregistered [ 869.725014] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 869.901910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.905610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.908889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 872.366635] Key type ._llcrypt registered [ 872.372061] Key type .llcrypt registered [ 872.499960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.508524] alg: No test for adler32 (adler32-zlib) [ 873.402139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.468310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 873.483124] LNet: Added LNI 192.168.206.47@tcp [8/512/0/180] [ 873.487042] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 873.488963] LNet: 52723:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 873.581740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.598652] LNet: 52742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.612491] LNetError: 52742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 873.619773] LNet: Removed LNI 10.1.2.3@tcp [ 873.622488] LNetError: 52742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.633650] LNet: Removed LNI 192.168.206.47@tcp [ 873.657125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.703779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 873.722615] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 873.797971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.816727] LNet: 52766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.821800] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.844958] LNet: Removed LNI 192.168.206.47@tcp [ 873.876450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.967035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 873.985603] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 873.991824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 873.993646] LNet: 52774:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 874.057226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.070734] LNet: 52792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.075317] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.080365] LNet: Removed LNI 10.1.2.3@tcp [ 874.082437] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.096510] LNet: Removed LNI 192.168.206.47@tcp [ 874.112191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.191822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 874.236847] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 874.242510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 874.244512] LNet: 52800:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 874.464414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.482536] LNet: 52826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.498841] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.510940] LNet: Removed LNI 10.1.2.3@tcp [ 874.530083] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.553776] LNet: Removed LNI 192.168.206.47@tcp [ 874.591546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.673696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 874.697756] LNet: Added LNI 192.168.206.47@tcp [8/256/0/182] [ 874.704699] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 874.809950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.846281] LNet: 52854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.855981] LNetError: 52854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.867576] LNet: Removed LNI 10.1.2.3@tcp [ 874.869610] LNetError: 52854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.897307] LNet: Removed LNI 192.168.206.47@tcp [ 874.924819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.011129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 875.041092] LNet: Added LNI 192.168.206.47@tcp [8/256/0/182] [ 875.056100] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 875.066628] LNet: 52863:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 875.153645] LNet: Removed LNI 10.1.2.3@tcp [ 875.172441] LNet: Removed LNI 192.168.206.47@tcp [ 875.785248] Key type .llcrypt unregistered [ 875.791140] Key type ._llcrypt unregistered [ 886.232393] Key type ._llcrypt registered [ 886.234268] Key type .llcrypt registered [ 887.451530] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 02:44:57 (1786689897) [ 888.722343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 890.428492] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:45:00 (1786689900) [ 890.772500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 890.788637] alg: No test for adler32 (adler32-zlib) [ 891.711941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.764089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.784269] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 892.440778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.47 [ 892.464244] LNet: 54130:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 892.471748] LNet: Removed LNI 192.168.206.47@tcp [ 898.160491] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:45:08 (1786689908) [ 898.252988] LNet: 54693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.258709] LNetError: 54693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.280781] LNet: Removed LNI 192.168.206.47@tcp [ 898.907616] Key type .llcrypt unregistered [ 898.921848] Key type ._llcrypt unregistered [ 904.680447] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 904.871142] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 907.490309] Key type ._llcrypt registered [ 907.495631] Key type .llcrypt registered [ 907.616308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 907.634601] alg: No test for adler32 (adler32-zlib) [ 908.787837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 913.587125] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:45:23 (1786689923) [ 913.719268] LNet: 55609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.731766] LNetError: 55609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 913.745821] LNet: Removed LNI 10.1.2.3@tcp [ 914.406561] Key type .llcrypt unregistered [ 914.409679] Key type ._llcrypt unregistered [ 920.566834] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.821506] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.826229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.831266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 923.600740] Key type ._llcrypt registered [ 923.602371] Key type .llcrypt registered [ 923.787939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.815197] alg: No test for adler32 (adler32-zlib) [ 924.749785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 930.378156] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:45:40 (1786689940) [ 931.566931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 933.178655] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:45:43 (1786689943) [ 933.299876] LNet: 56801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.311898] LNetError: 56801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.340835] LNet: Removed LNI 10.1.2.3@tcp [ 934.055969] Key type .llcrypt unregistered [ 934.058696] Key type ._llcrypt unregistered [ 940.230489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.418205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.423218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.430803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 943.133512] Key type ._llcrypt registered [ 943.137063] Key type .llcrypt registered [ 943.313784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.327369] alg: No test for adler32 (adler32-zlib) [ 944.290745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.365906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 951.922876] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:46:01 (1786689961) [ 952.343855] LNet: 58039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.347773] LNetError: 58039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.366407] LNet: Removed LNI 10.1.2.3@tcp [ 953.035650] Key type .llcrypt unregistered [ 953.038564] Key type ._llcrypt unregistered [ 957.965335] Key type ._llcrypt registered [ 957.967795] Key type .llcrypt registered [ 958.114069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.126439] alg: No test for adler32 (adler32-zlib) [ 959.027184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 959.060362] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 966.119863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 966.709910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_interrupt -r 1 [ 966.739360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_interrupt -r 1 [ 966.768173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 966.781815] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 966.983088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 967.028739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 967.073843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 967.111072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 967.143858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 967.172806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 967.200631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 967.233827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 967.253811] LNet: 58481:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.266789] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 967.280878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 967.330551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 967.363469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 967.402527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 967.442457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 967.473757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 967.507797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 968.026559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_dropped -r 1 [ 968.080293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_dropped -r 1 [ 968.119073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 968.158600] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 968.174155] LNet: Skipped 1 previous similar message [ 968.350835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 968.389963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 968.429593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 968.458780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 968.494642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 968.524483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 968.554296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 968.583898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 968.605778] LNet: 58590:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 968.624099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 968.648937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 968.672397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 968.701743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 968.735824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 968.767656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 968.800731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 969.316135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_aborted -r 1 [ 969.353690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_aborted -r 1 [ 969.387551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 969.403843] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 969.590197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 969.617834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 969.651494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 969.707294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 969.731920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 969.755146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 969.777881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 969.806064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 969.820647] LNet: 58699:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 969.848719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 969.899860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 969.946495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 969.971672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.997939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.032840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 970.065235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.504826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_no_route -r 1 [ 970.535365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_no_route -r 1 [ 970.567896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 970.797875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 970.826567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 970.855713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 970.887563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 970.920659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 970.949395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 970.975348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 971.000893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 971.021847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 971.040521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 971.062199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 971.084770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.109317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.141672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.174822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 971.603319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_timeout -r 1 [ 971.631944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_timeout -r 1 [ 971.660783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 971.675902] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 971.684841] LNet: Skipped 1 previous similar message [ 971.855715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 971.885147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 971.909252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 971.935920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 971.960567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 971.986939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 972.045342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 972.098491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 972.129738] LNet: 58918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 972.136055] LNet: 58918:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 972.158416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 972.192631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 972.238419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 972.268867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 972.316913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 972.346089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 972.392187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 972.805186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_error -r 1 [ 972.854919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_error -r 1 [ 972.903599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 973.114862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 973.143649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 973.172165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 973.205845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 973.237659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 973.272192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 973.305294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 973.332526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 973.369185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 973.398489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 973.430690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 973.459570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 973.494185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 973.518092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 973.541577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 973.808936] LNet: 59072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.816702] LNetError: 59072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.838200] LNet: Removed LNI 192.168.206.47@tcp [ 974.350832] Key type .llcrypt unregistered [ 974.357314] Key type ._llcrypt unregistered [ 992.589645] Key type ._llcrypt registered [ 992.591878] Key type .llcrypt registered [ 993.848975] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:46:43 (1786690003) [ 994.748211] Key type .llcrypt unregistered [ 994.751894] Key type ._llcrypt unregistered [ 1000.924145] Key type ._llcrypt registered [ 1000.926654] Key type .llcrypt registered [ 1001.094754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.117606] alg: No test for adler32 (adler32-zlib) [ 1002.056717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1002.098669] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1009.009347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1018.197630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.244803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1018.282665] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1019.687109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_interrupt -r 1 [ 1019.721820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 1 [ 1019.746945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_interrupt -r 1 [ 1019.778993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 1 [ 1019.818386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_interrupt -r 1 [ 1019.853899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 1 [ 1019.885990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_interrupt -r 1 [ 1019.917520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 1 [ 1019.947385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1019.968575] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1019.972347] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1020.190867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.226580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.257524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.301326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.333640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.380932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.425628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.459532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.485934] LNet: 60545:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1020.502945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.534759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.564920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.599287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.621756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.649422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.683712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1021.216480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_dropped -r 1 [ 1021.272408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 1 [ 1021.334202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_dropped -r 1 [ 1021.374545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 1 [ 1021.413117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_dropped -r 1 [ 1021.451131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 1 [ 1021.481601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_dropped -r 1 [ 1021.509863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 1 [ 1021.546898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1021.564582] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1021.567949] LNet: Skipped 2 previous similar messages [ 1021.753130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.778882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1021.810682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.832929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.858805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.888849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.914255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.938714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.969314] LNet: 60667:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1021.992703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.017838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.045651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.078558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.110532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.164687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.213579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1022.819639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_aborted -r 1 [ 1022.876471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 1 [ 1022.919518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_aborted -r 1 [ 1022.956502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 1 [ 1023.014972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_aborted -r 1 [ 1023.066895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 1 [ 1023.118024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_aborted -r 1 [ 1023.159758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 1 [ 1023.220086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1023.237669] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1023.252444] LNet: Skipped 2 previous similar messages [ 1023.473783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.509423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.536771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.567394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.593481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.619412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.663726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.689611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.709884] LNet: 60788:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.732956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.767888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.808295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.840150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.893202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.937074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.973072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1024.544693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_no_route -r 1 [ 1024.585217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 1 [ 1024.631268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_no_route -r 1 [ 1024.688148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 1 [ 1024.744819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_no_route -r 1 [ 1024.807889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 1 [ 1024.892428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_no_route -r 1 [ 1024.941384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 1 [ 1024.980515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1025.189693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.215884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.246251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.271949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.293437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.310441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.350799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.382452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.428760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.482797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.530284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.585671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.628605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.680443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.748306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1026.231769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_timeout -r 1 [ 1026.266711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 1 [ 1026.311795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_timeout -r 1 [ 1026.378531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 1 [ 1026.422997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_timeout -r 1 [ 1026.496983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 1 [ 1026.552772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_timeout -r 1 [ 1026.605747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 1 [ 1026.657383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1026.680141] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1026.687598] LNet: Skipped 5 previous similar messages [ 1026.940732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.977604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1027.017950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.056528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.108407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.171768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.201516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.225715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.243930] LNet: 61031:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.249045] LNet: 61031:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1027.270943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.302756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.345315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.385670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.428898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.471048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.500995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1028.095640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_error -r 1 [ 1028.123279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_error -r 1 [ 1028.179152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_error -r 1 [ 1028.206279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_error -r 1 [ 1028.238792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_error -r 1 [ 1028.280646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_error -r 1 [ 1028.323532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_error -r 1 [ 1028.364774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_error -r 1 [ 1028.414278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1028.639453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.673962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1028.713331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.763523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.810556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.856224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.884672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.936295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.985157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.024840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.063854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.098968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.142257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.185735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.233384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.404839] LNet: 61232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.411295] LNetError: 61232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.419395] LNet: Removed LNI 192.168.206.47@tcp [ 1034.432910] LNet: Removed LNI 192.168.206.47@tcp1 [ 1034.810552] Key type .llcrypt unregistered [ 1034.812791] Key type ._llcrypt unregistered [ 1050.598790] Key type ._llcrypt registered [ 1050.600943] Key type .llcrypt registered [ 1051.737354] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:47:41 (1786690061) [ 1052.372698] Key type .llcrypt unregistered [ 1052.374795] Key type ._llcrypt unregistered [ 1058.306892] Key type ._llcrypt registered [ 1058.308914] Key type .llcrypt registered [ 1058.449539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.462878] alg: No test for adler32 (adler32-zlib) [ 1059.380467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1059.399159] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1066.611703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1067.344558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_dropped -r 1 [ 1067.405371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_dropped -r 1 [ 1067.451941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1067.475957] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1067.721818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.756315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.785877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.820245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.858374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.884258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.914476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.965567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.000089] LNet: 62487:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.019197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.044874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.067708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.096841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.150929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.209592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.240191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.807773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_error -r 1 [ 1068.868269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_error -r 1 [ 1068.910868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1068.934177] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1069.184791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.213705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.236395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.260183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.284469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.310317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.332833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.367279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.383166] LNet: 62596:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.408202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.470866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.519433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.556610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.592597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.632216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.676648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1070.319415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_timeout -r 1 [ 1070.348833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_timeout -r 1 [ 1070.377421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1070.391324] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1070.645698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.667984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1070.692191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.714373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.751859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.779125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.809649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.846442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.864534] LNet: 62705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.891152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.921650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.957686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.987756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.011218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.033815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.074627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1071.469096] LNet: 62750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.481390] LNetError: 62750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.501739] LNet: Removed LNI 192.168.206.47@tcp [ 1071.902426] Key type .llcrypt unregistered [ 1071.906437] Key type ._llcrypt unregistered [ 1087.491116] Key type ._llcrypt registered [ 1087.493758] Key type .llcrypt registered [ 1088.551723] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:48:18 (1786690098) [ 1089.446311] Key type .llcrypt unregistered [ 1089.450199] Key type ._llcrypt unregistered [ 1095.220873] Key type ._llcrypt registered [ 1095.222433] Key type .llcrypt registered [ 1095.523442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1095.537479] alg: No test for adler32 (adler32-zlib) [ 1096.453541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1096.477149] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1103.937833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1113.839162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1113.897955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1113.932910] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1115.677872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_dropped -r 1 [ 1115.712707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 1 [ 1115.740216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_dropped -r 1 [ 1115.765134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 1 [ 1115.791973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_dropped -r 1 [ 1115.836318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 1 [ 1115.892167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_dropped -r 1 [ 1115.958932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 1 [ 1116.042541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1116.070899] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1116.081897] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.147@tcp [ 1116.294020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.338215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.375863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.404728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.445444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.491842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.535680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.570341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.588736] LNet: 64223:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.603664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.642244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.691981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.730902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.753955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.775179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.802058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.852031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.413872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_error -r 1 [ 1117.469477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_error -r 1 [ 1117.510155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_error -r 1 [ 1117.539350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_error -r 1 [ 1117.573863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_error -r 1 [ 1117.610348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_error -r 1 [ 1117.651492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_error -r 1 [ 1117.697312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_error -r 1 [ 1117.774903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1117.802892] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1117.809567] LNet: Skipped 2 previous similar messages [ 1118.025207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.054954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.083874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.111610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.138259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.167310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.199603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.230296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.250585] LNet: 64346:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.270735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.313868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.348894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.378765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.413850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.465294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.516329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.547917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.052969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_timeout -r 1 [ 1119.106310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_timeout -r 1 [ 1119.163384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_timeout -r 1 [ 1119.241704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_timeout -r 1 [ 1119.308451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_timeout -r 1 [ 1119.336329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_timeout -r 1 [ 1119.368183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_timeout -r 1 [ 1119.399290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_timeout -r 1 [ 1119.429241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.147@tcp [ 1119.454719] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1119.466495] LNet: Skipped 1 previous similar message [ 1119.697509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.731337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.773321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.807952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.845771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.878383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.921847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.976427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.995343] LNet: 64470:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.014293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.058758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.085849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.119706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.149431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.187106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.219772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.241251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.983105] LNet: 64552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.995384] LNetError: 64552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.008215] LNet: Removed LNI 192.168.206.47@tcp [ 1125.049818] LNet: Removed LNI 192.168.206.47@tcp1 [ 1125.508608] Key type .llcrypt unregistered [ 1125.511356] Key type ._llcrypt unregistered [ 1139.355803] Key type ._llcrypt registered [ 1139.358284] Key type .llcrypt registered [ 1140.412808] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:49:10 (1786690150) [ 1141.817529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1143.374465] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:49:13 (1786690153) [ 1143.972734] Key type .llcrypt unregistered [ 1143.976145] Key type ._llcrypt unregistered [ 1148.541381] Key type ._llcrypt registered [ 1148.542809] Key type .llcrypt registered [ 1148.737857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.753193] alg: No test for adler32 (adler32-zlib) [ 1149.600584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.620261] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1155.422572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1156.047907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e network_timeout -r 1 [ 1156.100598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e network_timeout -r 1 [ 1156.136115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1156.164106] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1156.372533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1156.405405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1156.451469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1156.484185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1156.528861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1156.562594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1156.587625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1156.613659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1156.632692] LNet: 66099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1156.651032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1156.679725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1156.710103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1156.740348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1156.769175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.803735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.046705] LNet: 66142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.058081] LNetError: 66142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.077332] LNet: Removed LNI 192.168.206.47@tcp [ 1157.383426] Key type .llcrypt unregistered [ 1157.385657] Key type ._llcrypt unregistered [ 1173.394895] Key type ._llcrypt registered [ 1173.397079] Key type .llcrypt registered [ 1173.558925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1173.578859] alg: No test for adler32 (adler32-zlib) [ 1174.455236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1174.484517] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1181.346593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1189.622366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.674760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1189.703022] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1190.876575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e network_timeout -r 1 [ 1190.927120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e network_timeout -r 1 [ 1190.998327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e network_timeout -r 1 [ 1191.048374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e network_timeout -r 1 [ 1191.105439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e network_timeout -r 1 [ 1191.186857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e network_timeout -r 1 [ 1191.282903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e network_timeout -r 1 [ 1191.344727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e network_timeout -r 1 [ 1191.381180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1191.403568] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1191.695531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.728195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.752858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.780847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.808347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.831458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.865252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.885982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.902605] LNet: 67026:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.909434] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1191.915545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.916369] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.147@tcp [ 1191.933299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.954739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.975718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.014614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.039800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.228051] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1196.573979] LNet: 67104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.580253] LNetError: 67104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.585945] LNet: Removed LNI 192.168.206.47@tcp [ 1196.598299] LNet: Removed LNI 192.168.206.47@tcp1 [ 1196.946732] Key type .llcrypt unregistered [ 1196.951395] Key type ._llcrypt unregistered [ 1209.689442] Key type ._llcrypt registered [ 1209.692211] Key type .llcrypt registered [ 1210.669323] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:50:20 (1786690220) [ 1210.981515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.000276] alg: No test for adler32 (adler32-zlib) [ 1211.924790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.984868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1212.022557] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1212.695893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.714509] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1213.412698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1213.491328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1213.544761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1213.582769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1213.625638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1213.678365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1213.726947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.47@tcp [ 1213.770801] LNetError: 68203:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1213.983284] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1213.986050] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1215.007943] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1216.150073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1219.103423] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1219.612361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1219.639515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.695978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1219.717220] LNet: 68300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.725132] LNetError: 68300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.767200] LNet: Removed LNI 192.168.206.47@tcp [ 1220.773939] LNetError: 68300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.788320] LNet: Removed LNI 192.168.206.47@tcp1 [ 1220.829566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.909969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1220.940661] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1221.726818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1221.744499] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1222.517779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1222.550743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.615168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1222.663901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1222.692211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1222.725939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1222.768818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1222.817219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.47@tcp [ 1222.836974] LNetError: 68573:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1222.943665] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1225.202978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1228.065033] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1228.071069] LNet: Skipped 1 previous similar message [ 1233.084803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1233.129585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.168245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.149857] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:50:49 (1786690249) [ 1239.275097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1239.312849] LNet: 69276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.321385] LNetError: 69276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.329606] LNet: Removed LNI 192.168.206.47@tcp [ 1239.345328] LNet: Removed LNI 192.168.206.47@tcp1 [ 1239.386469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.432865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1239.449116] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1240.058550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1240.081379] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1240.867846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1240.928256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1240.957222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1240.983480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1241.025278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1241.054887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1241.103494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.47@tcp --health 0 [ 1241.200650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1241.439487] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.47@tcp [ 1242.470060] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1242.482875] LNet: Skipped 1 previous similar message [ 1257.264719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1257.376481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1257.430590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.462546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.47@tcp --health 500 [ 1257.543659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.160735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.219263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.250837] LNet: 69781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.257450] LNetError: 69781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.271923] LNetError: 69781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1264.286241] LNet: Removed LNI 192.168.206.47@tcp [ 1264.307379] LNet: Removed LNI 192.168.206.47@tcp1 [ 1264.338525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.389240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1264.408746] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1265.094970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1265.111543] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1265.673378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1265.713838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.762486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1265.785273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1265.822248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1265.853129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1265.890560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1265.930431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.47@tcp --health 0 [ 1266.399458] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1266.409349] LNet: Skipped 3 previous similar messages [ 1267.207636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1277.442462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1277.487801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.538109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.821935] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:51:33 (1786690293) [ 1284.086637] LNet: 70826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.092117] LNetError: 70826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.095669] LNetError: 70826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1284.099261] LNet: Removed LNI 192.168.206.47@tcp [ 1284.110636] LNet: Removed LNI 192.168.206.47@tcp1 [ 1284.665093] Key type .llcrypt unregistered [ 1284.672939] Key type ._llcrypt unregistered [ 1291.700015] Key type ._llcrypt registered [ 1291.703576] Key type .llcrypt registered [ 1291.971178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1291.997124] alg: No test for adler32 (adler32-zlib) [ 1292.873387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1292.915136] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1302.311578] Lustre: DEBUG MARKER: Initial discovery [ 1302.407763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.147@tcp [ 1304.867525] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1304.945478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1305.004946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.147@tcp [ 1305.040907] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1305.076890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1305.154918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.147@tcp -r 1 -m GET -l 3 [ 1305.210523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.147@tcp -r 1 -m GET -e local_error [ 1305.255289] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1305.280808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.147@tcp -r 1 -m PUT -l 6 [ 1306.849415] Lustre: DEBUG MARKER: Force oleg647-client.virtnet to discover 192.168.206.147@tcp (in background) [ 1306.920633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.147@tcp [ 1308.044955] Lustre: DEBUG MARKER: Force oleg647-server to discover 192.168.206.47@tcp [ 1309.983686] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1310.578504] Lustre: DEBUG MARKER: Wait for 71585 [ 1312.257782] Lustre: DEBUG MARKER: Finished wait on 71585 [ 1312.306769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1312.337250] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1312.339923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1312.628454] LNet: 72046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.635417] LNetError: 72046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.651961] LNet: Removed LNI 192.168.206.47@tcp [ 1313.149749] Key type .llcrypt unregistered [ 1313.151262] Key type ._llcrypt unregistered [ 1328.162307] Key type ._llcrypt registered [ 1328.164397] Key type .llcrypt registered [ 1329.344688] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:52:19 (1786690339) [ 1329.825345] Key type .llcrypt unregistered [ 1329.830614] Key type ._llcrypt unregistered [ 1335.251978] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1335.377425] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1335.379903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1335.384145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1337.876704] Key type ._llcrypt registered [ 1337.884079] Key type .llcrypt registered [ 1338.044626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1338.066282] alg: No test for adler32 (adler32-zlib) [ 1338.967673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.010635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.027613] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1339.617578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1339.644157] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1346.529803] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:52:36 (1786690356) [ 1346.606438] LNet: 74119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.615371] LNetError: 74119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1346.623281] LNet: Removed LNI 10.1.2.3@tcp [ 1346.652289] LNet: Removed LNI 192.168.206.47@tcp [ 1347.229472] Key type .llcrypt unregistered [ 1347.231066] Key type ._llcrypt unregistered [ 1353.396394] Key type ._llcrypt registered [ 1353.399910] Key type .llcrypt registered [ 1353.571297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1353.585410] alg: No test for adler32 (adler32-zlib) [ 1354.467931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.526554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.556700] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1355.177730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1355.201772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1356.060942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.47@tcp [ 1357.151967] LNetError: Unexpected error -22 connecting to 192.168.206.47@tcp at host 192.168.206.47:988 [ 1361.644797] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:52:51 (1786690371) [ 1361.756514] LNet: 75256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.767737] LNetError: 75256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1361.787291] LNet: Removed LNI 10.1.2.3@tcp [ 1361.818722] LNet: Removed LNI 192.168.206.47@tcp [ 1362.435675] Key type .llcrypt unregistered [ 1362.437173] Key type ._llcrypt unregistered [ 1367.990842] Key type ._llcrypt registered [ 1367.993530] Key type .llcrypt registered [ 1368.193589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1368.209706] alg: No test for adler32 (adler32-zlib) [ 1369.105942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.151790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1369.176578] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1369.757311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1369.776996] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1370.652506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.47@tcp1 --nid 192.168.206.47@tcp2 [ 1374.751675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp1 192.168.206.47@tcp2 [ 1374.789832] LNetError: 76104:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.47@tcp1 to 192.168.206.47@tcp2 - Route table may be misconfigured [ 1374.819743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp2 192.168.206.47@tcp1 [ 1380.875748] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:53:10 (1786690390) [ 1381.093714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.112768] LNet: 76676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.120214] LNetError: 76676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.126629] LNet: Removed LNI 192.168.206.47@tcp1 [ 1382.183355] LNet: Removed LNI 192.168.206.47@tcp2 [ 1382.248918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.309920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1382.338710] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1382.948421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1382.982378] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1383.815722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1383.846474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.47@tcp [ 1383.871244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp1 192.168.206.47@tcp1 [ 1383.901706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp -d 192.168.206.47@tcp -e network_timeout [ 1383.928655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -e network_timeout [ 1383.958835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -e network_timeout [ 1383.989587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -e network_timeout [ 1384.018228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1384.032408] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1384.045475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1384.105072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.202127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1384.355665] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1389.546101] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:53:19 (1786690399) [ 1389.637280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.658363] LNet: 77531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.679923] LNetError: 77531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.687881] LNetError: 77531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1389.699052] LNet: Removed LNI 192.168.206.47@tcp [ 1390.766302] LNet: Removed LNI 192.168.206.47@tcp1 [ 1390.808902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.896962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1391.223220] LNet: 77579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.511712] Key type .llcrypt unregistered [ 1391.514536] Key type ._llcrypt unregistered [ 1400.904862] Key type ._llcrypt registered [ 1400.907339] Key type .llcrypt registered [ 1401.907297] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:53:31 (1786690411) [ 1402.256136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1402.268310] alg: No test for adler32 (adler32-zlib) [ 1403.206924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1404.364348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1406.127874] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:53:36 (1786690416) [ 1406.207458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.243736] LNet: 78668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.279642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.374805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.405301] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1407.004864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.020038] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1407.935677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1407.957891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp1 [ 1407.985557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp1 [ 1413.536349] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:53:43 (1786690423) [ 1415.195583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg647-server [ 1416.529319] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:53:46 (1786690426) [ 1418.253058] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg647-server [ 1419.789882] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:53:49 (1786690429) [ 1421.584157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg647-server [ 1423.378417] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:53:53 (1786690433) [ 1425.210856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg647-server [ 1427.047383] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:53:56 (1786690436) [ 1428.969833] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg647-server [ 1430.736193] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:54:00 (1786690440) [ 1432.366446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg647-server [ 1433.939634] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:54:03 (1786690443) [ 1435.838323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg647-server [ 1437.530896] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:54:07 (1786690447) [ 1439.284822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg647-server [ 1441.184923] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:54:10 (1786690450) [ 1443.213346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg647-server [ 1444.801600] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:54:14 (1786690454) [ 1444.889127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.918563] LNet: 82550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.927132] LNetError: 82550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.966091] LNet: Removed LNI 192.168.206.47@tcp [ 1446.007452] LNet: Removed LNI 192.168.206.47@tcp1 [ 1446.059601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.138953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.184539] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1446.903531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1446.988332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1447.200690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.231352] LNet: 82702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.251986] LNetError: 82702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.262895] LNetError: 82702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1447.288457] LNet: Removed LNI 192.168.206.47@tcp [ 1447.323700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1447.374483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1447.410673] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1448.325872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1448.567389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.601958] LNetError: 82847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.641910] LNet: Removed LNI 192.168.206.47@tcp [ 1448.657505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.722033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.747282] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1449.490683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1449.577829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1449.754801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.771333] LNet: 82994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.775156] LNet: 82994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1450.791290] LNet: Removed LNI 192.168.206.47@tcp [ 1450.811436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.866284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1450.897152] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1451.680557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1451.930979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.989604] LNetError: 83140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.005733] LNetError: 83140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1452.028426] LNet: Removed LNI 192.168.206.47@tcp [ 1452.060516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.116131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1452.150031] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1453.007919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1453.099376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1453.331408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.379437] LNet: Removed LNI 192.168.206.47@tcp [ 1454.422180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.502843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1454.579343] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1455.502842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1456.887161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.911348] LNet: 83442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.916018] LNet: 83442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1456.922166] LNetError: 83442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.928989] LNetError: 83442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1456.949292] LNet: Removed LNI 192.168.206.47@tcp [ 1456.993301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.054395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.073657] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1458.015923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1458.105916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1458.494510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.558061] LNet: Removed LNI 192.168.206.47@tcp [ 1459.601511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.668317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1459.721803] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1460.605201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1460.859166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.927735] LNet: Removed LNI 192.168.206.47@tcp [ 1461.953761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.022447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.061148] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1462.878240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1462.950121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1463.215962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.298798] LNet: Removed LNI 192.168.206.47@tcp [ 1464.317415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.353327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1464.371595] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1465.028268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1465.314415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.335562] LNet: 84027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.342991] LNet: 84027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1465.358208] LNetError: 84027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.367962] LNetError: 84027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1465.384406] LNet: Removed LNI 192.168.206.47@tcp [ 1465.399911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.448638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1465.491880] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1466.229869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1466.319495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1466.608373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.698232] LNet: Removed LNI 192.168.206.47@tcp [ 1467.728296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.800224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1467.839116] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1468.877270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1469.141227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1470.182747] LNet: Removed LNI 192.168.206.47@tcp [ 1470.213529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.330987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1470.371467] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1471.050525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1471.161368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.203682] LNet: Removed LNI 192.168.206.47@tcp [ 1471.225476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.309338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.345912] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1472.205164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1472.282375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.315949] LNet: Removed LNI 192.168.206.47@tcp [ 1472.369388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.432522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1472.514047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.554803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.610404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1472.711881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.772566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.844977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1472.961603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.994423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.057298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1473.084738] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1473.956486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.004519] LNet: Removed LNI 192.168.206.47@tcp [ 1474.041554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.139481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1482.424460] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:54:52 (1786690492) [ 1482.591730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.633985] LNet: 85335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.650084] LNet: 85335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1482.697569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.834140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1482.879960] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1484.033294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.050978] LNetError: 85480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.055696] LNetError: 85480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1484.069992] LNet: Removed LNI 192.168.206.47@tcp231 [ 1484.083353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.137773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1484.153755] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1484.160963] LNet: 85488:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1484.584298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1484.623677] LNet: Removed LNI 192.168.206.47@tcp231 [ 1484.641663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1484.662309] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1486.250627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.282137] LNet: Removed LNI 192.168.206.47@tcp231 [ 1486.308277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.443329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1486.473527] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/0] [ 1486.478307] LNet: 85640:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.889775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1486.945820] LNet: Removed LNI 192.168.206.47@tcp231 [ 1486.965177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1487.007425] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/0] [ 1488.059509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.080444] LNet: Removed LNI 192.168.206.47@tcp231 [ 1488.093587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.166424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1488.204774] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/60] [ 1488.217759] LNet: 85792:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1488.571837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1488.619545] LNet: Removed LNI 192.168.206.47@tcp231 [ 1488.647844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1488.680159] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/60] [ 1489.935624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.980991] LNet: Removed LNI 192.168.206.47@tcp231 [ 1490.020590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.106871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1490.150852] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1490.582910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1490.649177] LNet: Removed LNI 192.168.206.47@tcp231 [ 1490.676610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1490.726736] LNet: Added LNI 192.168.206.47@tcp231 [8/256/0/180] [ 1497.330337] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:55:07 (1786690507) [ 1497.652687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.693466] LNet: Removed LNI 192.168.206.47@tcp231 [ 1497.729204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.826923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1497.868270] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1498.710328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1498.744410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1498.783620] LNet: Removed LNI 192.168.206.47@tcp232 [ 1498.804062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1498.826705] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1499.671894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1499.717804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1499.747413] LNet: Removed LNI 192.168.206.47@tcp232 [ 1499.766562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1499.789967] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1500.575243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1500.605343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1500.629697] LNet: Removed LNI 192.168.206.47@tcp232 [ 1500.641521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1500.660289] LNet: Added LNI 192.168.206.47@tcp232 [8/256/0/180] [ 1501.239405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1501.279059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1501.312442] LNet: Removed LNI 192.168.206.47@tcp232 [ 1501.354997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1506.652699] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:55:16 (1786690516) [ 1507.531263] Key type .llcrypt unregistered [ 1507.533367] Key type ._llcrypt unregistered [ 1512.581929] Key type ._llcrypt registered [ 1512.584475] Key type .llcrypt registered [ 1512.715991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1512.725806] alg: No test for adler32 (adler32-zlib) [ 1513.593820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1513.618980] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1520.345253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1529.259343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.316490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1529.333859] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1530.600167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_interrupt -r 2 [ 1530.678209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 2 [ 1530.727344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_interrupt -r 2 [ 1530.759960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 2 [ 1530.797304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_interrupt -r 2 [ 1530.835610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_interrupt -r 2 [ 1530.868447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_interrupt -r 2 [ 1530.920575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_interrupt -r 2 [ 1530.958926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1530.990986] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1530.997982] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1531.212123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1531.237556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1531.265653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1531.292734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1531.325595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1531.350184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1531.383428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1531.428779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1531.445142] LNet: 88449:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1531.462407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1531.497711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1531.530878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1531.579212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1531.617094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1531.661760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1532.249757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_dropped -r 2 [ 1532.286294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 2 [ 1532.333546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_dropped -r 2 [ 1532.382739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 2 [ 1532.412797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_dropped -r 2 [ 1532.448677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_dropped -r 2 [ 1532.489873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_dropped -r 2 [ 1532.529488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_dropped -r 2 [ 1532.566767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1532.589952] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1532.824290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1532.846588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1532.868719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1532.889466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1532.915041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1532.943132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1532.966925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1532.996175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1533.012718] LNet: 88569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1533.025577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1533.058455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1533.088988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1533.123529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1533.186456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1533.220500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1533.711229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_aborted -r 2 [ 1533.754729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 2 [ 1533.787664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_aborted -r 2 [ 1533.813608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 2 [ 1533.854237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_aborted -r 2 [ 1533.899680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_aborted -r 2 [ 1533.928366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_aborted -r 2 [ 1533.957716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_aborted -r 2 [ 1533.996583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1534.019269] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1534.287525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1534.321083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1534.348879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1534.375279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1534.403523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1534.436821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1534.476452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1534.507695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1534.524783] LNet: 88690:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1534.552419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1534.588647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1534.622898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1534.662636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1534.705476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1534.739328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1535.299742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_no_route -r 2 [ 1535.338501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 2 [ 1535.393066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_no_route -r 2 [ 1535.479040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 2 [ 1535.537862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_no_route -r 2 [ 1535.590791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_no_route -r 2 [ 1535.626905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_no_route -r 2 [ 1535.669931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_no_route -r 2 [ 1535.702964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1535.998475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1536.039780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1536.072703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1536.113789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1536.137404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1536.173484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1536.218491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1536.256595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1536.296559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1536.331842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1536.388673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1536.457643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1536.505713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1536.563704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1537.174425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e local_timeout -r 2 [ 1537.229371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 2 [ 1537.292985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e local_timeout -r 2 [ 1537.336735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 2 [ 1537.366868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e local_timeout -r 2 [ 1537.421685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e local_timeout -r 2 [ 1537.450494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e local_timeout -r 2 [ 1537.483618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e local_timeout -r 2 [ 1537.516670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1537.532888] LNet: There was an unexpected network error while writing to 192.168.206.147: rc = -22 [ 1537.542960] LNet: Skipped 1 previous similar message [ 1537.735771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1537.758151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1537.778688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1537.801605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1537.822722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1537.844642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1537.868512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1537.889406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1537.901455] LNet: 88930:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1537.904719] LNet: 88930:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1537.916346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1537.938413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1537.961359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1537.984861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1538.010840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1538.038065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1538.463536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp -m GET -e remote_dropped -r 2 [ 1538.488707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 2 [ 1538.512225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp -m GET -e remote_dropped -r 2 [ 1538.541662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 2 [ 1538.568734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp -m GET -e remote_dropped -r 2 [ 1538.615829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.147@tcp1 -m GET -e remote_dropped -r 2 [ 1538.667935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp -m GET -e remote_dropped -r 2 [ 1538.703789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.47@tcp1 -d 192.168.206.47@tcp1 -m GET -e remote_dropped -r 2 [ 1538.733903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.47@tcp 192.168.206.147@tcp [ 1538.747817] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.147@tcp [ 1538.945557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1538.975937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1539.032756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1539.079501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1539.106457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1539.132124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1539.161718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1539.191760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1539.250606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1539.308619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1539.351196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1539.391319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1539.427321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1539.471873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1543.806281] LNet: 89128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.812768] LNetError: 89128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.821181] LNet: Removed LNI 192.168.206.47@tcp [ 1543.840628] LNet: Removed LNI 192.168.206.47@tcp1 [ 1544.268715] Key type .llcrypt unregistered [ 1544.272661] Key type ._llcrypt unregistered [ 1559.697989] Key type ._llcrypt registered [ 1559.700878] Key type .llcrypt registered [ 1560.963028] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:56:10 (1786690570) [ 1561.454126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.466360] alg: No test for adler32 (adler32-zlib) [ 1562.356296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.407960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.437750] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1563.104926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1563.128732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.47@tcp [ 1563.143427] LNetError: 90107:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1563.215649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.47@tcp [ 1563.305626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1569.059489] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:56:19 (1786690579) [ 1569.198484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.217069] LNet: 90694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.226212] LNetError: 90694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.248057] LNet: Removed LNI 192.168.206.47@tcp [ 1569.308988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.362207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.381878] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1570.070145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1570.084176] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1570.970048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.47@tcp [ 1571.007399] LNetError: 90963:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.47@tcp to recovery [ 1571.423318] LNet: 1 local NIs in recovery (showing 1): 192.168.206.47@tcp [ 1571.424308] LNet: There was an unexpected network error while writing to 192.168.206.47: rc = -22 [ 1572.285519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.47@tcp [ 1579.603843] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:56:29 (1786690589) [ 1581.176755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg647-server [ 1582.710552] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:56:32 (1786690592) [ 1584.344135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg647-server [ 1585.965956] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:56:36 (1786690596) [ 1587.850172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg647-server [ 1589.588705] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:56:39 (1786690599) [ 1589.711883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.741432] LNet: 92590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.760159] LNetError: 92590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.771434] LNet: Removed LNI 192.168.206.47@tcp [ 1589.797473] LNet: Removed LNI 192.168.206.47@tcp1 [ 1589.810864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.852804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.871620] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1590.632773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1590.724266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1590.790567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1590.869231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1590.910642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1590.941635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1596.624358] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:56:46 (1786690606) [ 1596.703568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.728447] LNet: 93311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.736781] LNetError: 93311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.743675] LNetError: 93311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1596.771019] LNet: Removed LNI 192.168.206.47@tcp [ 1596.810678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.863385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.890866] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1602.884786] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:56:52 (1786690612) [ 1603.035391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.059088] LNet: 94021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.068180] LNetError: 94021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.081362] LNet: Removed LNI 192.168.206.47@tcp [ 1603.105661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.147941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.180436] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1609.546401] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:56:59 (1786690619) [ 1610.965956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1612.760626] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:57:02 (1786690622) [ 1613.206770] LNet: 95055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.216637] LNetError: 95055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.243370] LNet: Removed LNI 192.168.206.47@tcp [ 1613.951765] Key type .llcrypt unregistered [ 1613.956857] Key type ._llcrypt unregistered [ 1619.755749] Key type ._llcrypt registered [ 1619.761087] Key type .llcrypt registered [ 1620.049933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1620.072532] alg: No test for adler32 (adler32-zlib) [ 1621.049357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1621.090560] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1628.167088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1633.183523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.147@tcp [ 1633.484558] LNet: 95481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.495757] LNetError: 95481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.526401] LNet: Removed LNI 192.168.206.47@tcp [ 1634.082413] Key type .llcrypt unregistered [ 1634.084394] Key type ._llcrypt unregistered [ 1645.398185] Key type ._llcrypt registered [ 1645.401287] Key type .llcrypt registered [ 1646.608897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1648.245842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1649.899563] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:57:39 (1786690659) [ 1651.218460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1652.664269] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:57:42 (1786690662) [ 1654.106883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg647-server [ 1655.535992] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:57:45 (1786690665) [ 1657.243355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg647-server [ 1658.887243] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:57:48 (1786690668) [ 1660.271180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1660.313151] alg: No test for adler32 (adler32-zlib) [ 1661.424175] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 1661.646200] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1663.337907] Key type lgssc registered [ 1664.446654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1683.637249] Key type lgssc unregistered [ 1683.902550] LNet: 98658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.913517] LNetError: 98658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.931063] LNet: Removed LNI 192.168.206.47@tcp [ 1684.539491] Key type .llcrypt unregistered [ 1684.543748] Key type ._llcrypt unregistered [ 1699.043515] Key type ._llcrypt registered [ 1699.050442] Key type .llcrypt registered [ 1700.175621] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:58:30 (1786690710) [ 1700.588296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.599497] alg: No test for adler32 (adler32-zlib) [ 1701.498780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.577469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1701.639609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1701.680535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1701.716953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1708.072330] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:58:38 (1786690718) [ 1708.211578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.281492] LNet: 100070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.315314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.375401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.402972] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1709.112658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.127613] LNet: 100217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.132492] LNetError: 100217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.150276] LNet: Removed LNI 192.168.206.47@tcp [ 1709.175397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.243438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1709.269262] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1709.868222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.908715] LNetError: 100245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.928563] LNet: Removed LNI 192.168.206.47@tcp [ 1709.964124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.019329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1710.044105] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1716.305759] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:58:46 (1786690726) [ 1716.384414] LNet: 100826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.394733] LNet: 100826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1716.409386] LNetError: 100826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.450297] LNet: Removed LNI 192.168.206.47@tcp [ 1717.055564] Key type .llcrypt unregistered [ 1717.056981] Key type ._llcrypt unregistered [ 1723.867586] Key type ._llcrypt registered [ 1723.870207] Key type .llcrypt registered [ 1724.140294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1724.175211] alg: No test for adler32 (adler32-zlib) [ 1725.141174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.225065] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1725.227211] LNet: *** cfs_fail_loc=e003, val=100*** [ 1726.021979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1726.058689] LNet: *** cfs_fail_loc=e003, val=100*** [ 1726.060620] LNet: Skipped 2 previous similar messages [ 1726.079441] Ping reply received with mapping: [ 1726.082102] found metadata reply NI[192.168.206.47@tcp] [ 1726.094633] 2f ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1726.104457] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1726.109753] 5a 5a 5a 5a 5a 5a 5a 5a [ 1726.281292] LNet: 101237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.303386] LNetError: 101237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.328749] LNet: Removed LNI 192.168.206.47@tcp [ 1727.096427] Key type .llcrypt unregistered [ 1727.100247] Key type ._llcrypt unregistered [ 1739.010484] Key type ._llcrypt registered [ 1739.013445] Key type .llcrypt registered [ 1740.310589] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:59:10 (1786690750) [ 1740.833422] Key type .llcrypt unregistered [ 1740.835838] Key type ._llcrypt unregistered [ 1747.587480] Key type ._llcrypt registered [ 1747.593273] Key type .llcrypt registered [ 1747.915555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.945416] alg: No test for adler32 (adler32-zlib) [ 1749.153245] LNetError: 102294:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1749.537413] Key type .llcrypt unregistered [ 1749.544237] Key type ._llcrypt unregistered [ 1754.300684] Key type ._llcrypt registered [ 1754.305418] Key type .llcrypt registered [ 1755.326734] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:59:25 (1786690765) [ 1755.753318] Key type .llcrypt unregistered [ 1755.757436] Key type ._llcrypt unregistered [ 1761.060533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1761.065599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1763.531843] Key type ._llcrypt registered [ 1763.533525] Key type .llcrypt registered [ 1763.725648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1763.739690] alg: No test for adler32 (adler32-zlib) [ 1764.663382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.748373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1773.079121] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:59:43 (1786690783) [ 1773.269696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.306871] LNet: 104082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.320638] LNetError: 104082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1773.342190] LNet: Removed LNI 10.1.2.3@tcp [ 1773.364482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.073160] Key type .llcrypt unregistered [ 1774.078016] Key type ._llcrypt unregistered [ 1779.913143] Key type ._llcrypt registered [ 1779.915802] Key type .llcrypt registered [ 1780.160394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1780.184298] alg: No test for adler32 (adler32-zlib) [ 1781.156826] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1781.236906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1781.389274] LNet: 104375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.394756] LNetError: 104375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.414209] LNet: Removed LNI 192.168.206.47@tcp [ 1782.033488] Key type .llcrypt unregistered [ 1782.036251] Key type ._llcrypt unregistered [ 1792.417856] Key type ._llcrypt registered [ 1792.422500] Key type .llcrypt registered [ 1793.532093] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 03:00:03 (1786690803) [ 1794.302410] Key type .llcrypt unregistered [ 1794.306199] Key type ._llcrypt unregistered [ 1800.203764] Key type ._llcrypt registered [ 1800.205959] Key type .llcrypt registered [ 1800.356668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1800.371974] alg: No test for adler32 (adler32-zlib) [ 1806.851895] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 03:00:16 (1786690816) [ 1806.958796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.000140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.080785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.168748] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1807.844324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.876539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1807.904667] LNet: Removed LNI 192.168.206.47@tcp [ 1808.186933] LNet: 106130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.552230] Key type .llcrypt unregistered [ 1808.557975] Key type ._llcrypt unregistered [ 1818.440420] Key type ._llcrypt registered [ 1818.443354] Key type .llcrypt registered [ 1819.584680] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 03:00:29 (1786690829) [ 1820.010397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.028631] alg: No test for adler32 (adler32-zlib) [ 1820.920384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.973659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.002106] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1821.545463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1821.578180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.47@tcp [ 1821.600890] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1821.604809] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.608936] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 0 [ 1821.620219] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1821.624214] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1821.630969] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.637587] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 0 [ 1821.644194] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1821.648188] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1821.651861] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.657372] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 1871 [ 1821.667499] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1821.672611] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1821.677971] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1821.681162] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1821, 0, 1871 [ 1821.685407] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1826.743329] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 03:00:37 (1786690837) [ 1827.154555] LNet: 107670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.166563] LNetError: 107670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.200560] LNet: Removed LNI 192.168.206.47@tcp [ 1828.914358] Key type .llcrypt unregistered [ 1828.916430] Key type ._llcrypt unregistered [ 1834.306785] Key type ._llcrypt registered [ 1834.311702] Key type .llcrypt registered [ 1834.521663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1834.548228] alg: No test for adler32 (adler32-zlib) [ 1835.504758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1835.532952] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1842.048614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 1849.749592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.818835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1849.848177] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 1851.237816] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1851.364384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1851.367886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1851.371228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1853.712260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1853.747281] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1864.384669] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 03:01:14 (1786690874) [ 1864.511514] LNet: 109182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.526339] LNetError: 109182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.534384] LNet: Removed LNI 192.168.206.47@tcp [ 1865.569858] LNet: Removed LNI 192.168.206.47@tcp1 [ 1865.578265] LNetError: 109182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1865.592472] LNetError: 109182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1865.629425] LNet: Removed LNI 10.1.2.3@tcp99 [ 1866.410379] Key type .llcrypt unregistered [ 1866.413230] Key type ._llcrypt unregistered [ 1873.259798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1873.269060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1875.720265] Key type ._llcrypt registered [ 1875.721875] Key type .llcrypt registered [ 1875.874582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1875.883917] alg: No test for adler32 (adler32-zlib) [ 1876.817273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.867975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1876.895518] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1877.409500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1877.431914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1879.343441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1879.396784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1879.464856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1879.487084] LNetError: 109938:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1879.533507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1879.579662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1879.904764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1879.981078] LNet: 109958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.997201] LNetError: 109958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1880.005972] LNet: Removed LNI 10.1.2.3@tcp [ 1881.066144] LNet: Removed LNI 192.168.206.47@tcp [ 1881.858254] Key type .llcrypt unregistered [ 1881.861391] Key type ._llcrypt unregistered [ 1891.946832] Key type ._llcrypt registered [ 1891.949072] Key type .llcrypt registered [ 1892.986366] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 03:01:43 (1786690903) [ 1893.285972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1893.295886] alg: No test for adler32 (adler32-zlib) [ 1894.207970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.247970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.265454] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1894.809627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1894.864024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg647-client@tcp [ 1899.201371] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 03:01:49 (1786690909) [ 1899.339815] LNet: 111493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.346580] LNetError: 111493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.367431] LNet: Removed LNI 192.168.206.47@tcp [ 1899.798686] Key type .llcrypt unregistered [ 1899.800979] Key type ._llcrypt unregistered [ 1904.068278] Key type ._llcrypt registered [ 1904.070803] Key type .llcrypt registered [ 1904.219191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.231363] alg: No test for adler32 (adler32-zlib) [ 1905.090289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1905.105841] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1905.123939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1909.910633] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 03:02:00 (1786690920) [ 1911.046689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1912.184358] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 03:02:02 (1786690922) [ 1913.275731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1914.421598] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 03:02:04 (1786690924) [ 1914.524265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.541068] LNet: 112915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.549363] LNetError: 112915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.557538] LNet: Removed LNI 192.168.206.47@tcp [ 1914.569248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.604611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.630195] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1915.222840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1915.257055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1919.627398] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 03:02:09 (1786690929) [ 1919.758356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.777748] LNet: 113625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.782903] LNetError: 113625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.807199] LNet: Removed LNI 192.168.206.47@tcp [ 1920.844182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.902085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1920.939465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1921.159772] LNet: 113666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.393368] Key type .llcrypt unregistered [ 1921.396090] Key type ._llcrypt unregistered [ 1930.320264] Key type ._llcrypt registered [ 1930.322848] Key type .llcrypt registered [ 1931.341339] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 03:02:21 (1786690941) [ 1931.640760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.654973] alg: No test for adler32 (adler32-zlib) [ 1932.571463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.644072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.672805] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1933.242510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1933.290313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1938.513817] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 03:02:28 (1786690948) [ 1938.673552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.695403] LNet: 115183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.704391] LNetError: 115183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.733382] LNet: Removed LNI 192.168.206.47@tcp [ 1938.761419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.813827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1938.851349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1943.726331] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 03:02:34 (1786690954) [ 1943.822277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.842236] LNet: 115759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.864477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.942288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1944.242692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1944.286076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1944.343975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1944.577944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1944.601204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1944.659278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.694645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.762265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1945.077533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1945.107595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1945.155308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1945.468480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1945.500819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1950.386471] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 03:02:40 (1786690960) [ 1950.521275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.554405] LNet: 116399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.561830] LNet: 116399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1950.585435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.650461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.670036] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1951.419587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1951.446366] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1952.270609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1952.328826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1952.507512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.520785] LNetError: 116678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.532987] LNet: Removed LNI 192.168.206.47@tcp [ 1952.547355] LNet: Removed LNI 192.168.206.47@tcp2 [ 1952.609604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.687172] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1952.691882] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1952.700838] LNet: 116684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1953.176757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1953.339821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.367439] LNet: 116721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.375696] LNet: 116721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1954.399188] LNet: Removed LNI 192.168.206.47@tcp [ 1954.403169] LNetError: 116721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.413550] LNetError: 116721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1954.434388] LNet: Removed LNI 192.168.206.47@tcp2 [ 1954.455623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.519415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1954.539777] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1955.138859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1955.159272] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1955.876673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1955.914156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1955.977177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1956.202952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.279140] LNet: Removed LNI 192.168.206.47@tcp [ 1957.282475] LNetError: 117005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.288497] LNetError: 117005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1957.306355] LNet: Removed LNI 192.168.206.47@tcp2 [ 1957.326854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.387375] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1957.393253] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1957.398403] LNet: 117012:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1957.813732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1957.963771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1957.982409] LNet: 117049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.987186] LNet: 117049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1957.992615] LNet: Removed LNI 192.168.206.47@tcp [ 1959.021841] LNet: Removed LNI 192.168.206.47@tcp2 [ 1959.056474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.107701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1959.129565] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1959.820234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1959.840667] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1960.531610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1960.575387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1960.735693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.759354] LNet: Removed LNI 192.168.206.47@tcp [ 1961.763095] LNetError: 117328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.768192] LNetError: 117328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1961.779796] LNet: Removed LNI 192.168.206.47@tcp2 [ 1961.802865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.866412] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1961.872521] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1961.878696] LNet: 117334:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1962.371061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1962.581404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.615140] LNet: Removed LNI 192.168.206.47@tcp [ 1963.622990] LNet: Removed LNI 192.168.206.47@tcp2 [ 1963.649722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.706826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.724154] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1964.405055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1964.425438] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1965.113303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1965.140647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1965.185417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1965.366957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.431138] LNet: Removed LNI 192.168.206.47@tcp [ 1967.467829] LNet: Removed LNI 192.168.206.47@tcp2 [ 1967.502295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.562622] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1967.567272] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1967.577357] LNet: 117662:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1968.095531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1968.280601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.295956] LNet: 117699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.300959] LNet: 117699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1968.308770] LNet: Removed LNI 192.168.206.47@tcp [ 1969.381775] LNet: Removed LNI 192.168.206.47@tcp2 [ 1969.424784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.475561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.500360] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1970.102887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1970.131595] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1970.799093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1970.826283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1970.881880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1971.156457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1971.177426] LNetError: 117996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.185457] LNetError: 117996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1971.190458] LNet: Removed LNI 192.168.206.47@tcp [ 1971.200589] LNet: Removed LNI 192.168.206.47@tcp2 [ 1971.222992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.289321] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1971.296606] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1971.734352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1972.008067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.023211] LNet: Removed LNI 192.168.206.47@tcp [ 1973.036150] LNet: Removed LNI 192.168.206.47@tcp2 [ 1973.059499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.105714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.129981] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1973.703180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1973.716383] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1974.444851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1974.476362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1974.499833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1974.553801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1974.859460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.903547] LNet: Removed LNI 192.168.206.47@tcp [ 1976.943387] LNet: Removed LNI 192.168.206.47@tcp2 [ 1976.995724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.051267] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1977.057353] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 1977.062228] LNet: 118362:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1977.068209] LNet: 118362:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1977.735762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 1983.953058] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1985.140660] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 03:03:15 (1786690995) [ 1985.249825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.284237] LNet: 119118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.290775] LNet: 119118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1985.301473] LNet: Removed LNI 192.168.206.47@tcp [ 1986.344875] LNet: Removed LNI 192.168.206.47@tcp2 [ 1986.370770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.594705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1986.610814] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1987.310548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1987.327717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1987.990812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.001763] LNetError: 119395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1988.008229] LNetError: 119395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1988.012335] LNet: Removed LNI 10.1.2.3@tcp [ 1988.022271] LNet: Removed LNI 192.168.206.47@tcp [ 1988.038365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.094238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1988.113240] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1988.117153] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1988.477909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1988.501516] LNet: Removed LNI 192.168.206.47@tcp [ 1988.515740] LNet: Removed LNI 10.1.2.3@tcp [ 1993.131503] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 03:03:23 (1786691003) [ 1993.310172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.336848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.374402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.391979] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 1994.077791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1994.104995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.139941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.185420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.220644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.258996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.295303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.324428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.346961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.382698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.407318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.47@tcp [ 1994.518814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2000.219236] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 03:03:30 (1786691010) [ 2000.301550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.382596] LNet: Removed LNI 192.168.206.47@tcp [ 2001.411622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.557943] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2001.664822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2001.668943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2001.673159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2004.007841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.037121] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2004.680859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2004.709945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2006.360359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2006.401851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2006.462647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.47@tcp -m PUT -e local_timeout -r 1 [ 2006.531912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2006.588765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2006.592503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2006.849803] LNetError: 120802:0:(lib-move.c:4801:lnet_parse()) 192.168.206.47@tcp, src 192.168.206.47@tcp: Dropping PUT (error -108 looking up sender) [ 2007.909556] LNet: Removed LNI 10.1.2.3@tcp [ 2008.936435] LNet: Removed LNI 192.168.206.47@tcp [ 2009.361626] Key type .llcrypt unregistered [ 2009.367847] Key type ._llcrypt unregistered [ 2020.740761] Key type ._llcrypt registered [ 2020.747985] Key type .llcrypt registered [ 2022.064731] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 03:03:51 (1786691031) [ 2022.401945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2022.420794] alg: No test for adler32 (adler32-zlib) [ 2023.328811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.747312] LNet: 122120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.483387] Key type .llcrypt unregistered [ 2024.486713] Key type ._llcrypt unregistered [ 2030.875517] Key type ._llcrypt registered [ 2030.880356] Key type .llcrypt registered [ 2031.074411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2031.088283] alg: No test for adler32 (adler32-zlib) [ 2032.054391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2032.077815] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2038.507810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 2038.731385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2039.269770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.147@tcp [ 2039.477912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2039.506963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2039.541703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2039.595673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2039.637540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2039.664821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2039.689398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2039.717622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2039.731284] LNet: 122575:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2039.742974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2039.765985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2039.794256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2039.818339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2039.839441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2039.862663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2040.392358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2040.560423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2040.595805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2040.622886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2040.646659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2040.682310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2040.721800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2040.753705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2040.780692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2040.798241] LNet: 122693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2040.818419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2040.842713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2040.870879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2040.907701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2040.935914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2040.964556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2041.459870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.147@tcp [ 2041.661535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2041.685722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2041.713659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2041.735460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2041.763378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2041.787341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2041.813330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2041.836773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2041.848596] LNet: 122810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2041.869735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2041.895446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2041.921973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2041.945302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2041.977576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2042.024773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2042.531130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2042.745329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2042.777778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2042.812495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2042.839792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2042.861405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2042.883645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2042.909907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2042.937224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2042.958753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2042.979756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2043.002753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2043.025808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2043.060888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2043.089858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2043.435434] LNet: 122977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.445641] LNetError: 122977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.459843] LNet: Removed LNI 192.168.206.47@tcp [ 2043.811714] Key type .llcrypt unregistered [ 2043.817390] Key type ._llcrypt unregistered [ 2058.025506] Key type ._llcrypt registered [ 2058.027838] Key type .llcrypt registered [ 2059.124709] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 03:04:29 (1786691069) [ 2059.498455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2059.523222] alg: No test for adler32 (adler32-zlib) [ 2060.476986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.781629] LNet: 123865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.371654] Key type .llcrypt unregistered [ 2061.374724] Key type ._llcrypt unregistered [ 2067.419963] Key type ._llcrypt registered [ 2067.422792] Key type .llcrypt registered [ 2067.628480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2067.640431] alg: No test for adler32 (adler32-zlib) [ 2068.576467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2068.617668] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2075.935360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.147@tcp [ 2083.967338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.014796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2084.037579] LNet: Added LNI 192.168.206.47@tcp1 [8/256/0/180] [ 2084.806267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2085.392287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.147@tcp [ 2085.605272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2085.636129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2085.669907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2085.695670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2085.722623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2085.756590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2085.784844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2085.820418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2085.843308] LNet: 124521:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2085.862719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2085.909987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2085.940801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2085.975600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2086.004729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2086.032679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2086.513331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2086.693598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2086.738188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2086.763584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2086.797023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2086.837644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2086.873812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2086.903079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2086.928199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2086.950667] LNet: 124639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2086.986688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2087.026756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2087.069526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2087.076323] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.147@tcp, 192.168.206.147@tcp1 [ 2087.124377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2087.171323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2087.207506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2087.826342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.147@tcp [ 2088.072237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2088.110602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2088.138509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2088.167864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2088.192383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2088.221702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2088.249641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2088.283650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2088.300822] LNet: 124756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2088.315502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2088.337571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2088.360130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2088.384297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2088.406967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2088.450708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2088.911634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2089.108462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2089.148083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2089.198302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2089.245322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2089.272596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2089.307808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2089.336854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2089.370463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2089.398521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2089.426325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2089.459489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2089.487207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2089.521553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2089.561875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2094.038768] LNet: 124958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.047340] LNetError: 124958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.056792] LNet: Removed LNI 192.168.206.47@tcp [ 2094.066124] LNet: Removed LNI 192.168.206.47@tcp1 [ 2094.498576] Key type .llcrypt unregistered [ 2094.505433] Key type ._llcrypt unregistered [ 2108.877211] Key type ._llcrypt registered [ 2108.880099] Key type .llcrypt registered [ 2109.911174] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 03:05:20 (1786691120) [ 2110.259906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.280066] alg: No test for adler32 (adler32-zlib) [ 2111.173787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.232318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2111.251086] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2111.756801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2111.774265] LNet: Added LNI 192.168.206.47@tcp2 [8/256/0/180] [ 2112.615671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.47@tcp [ 2112.698694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2113.759156] LNet: Removed LNI 192.168.206.47@tcp [ 2119.335477] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 03:05:29 (1786691129) [ 2120.606058] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2122.036634] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 03:05:32 (1786691132) [ 2127.970243] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 03:05:37 (1786691137) [ 2133.593582] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 03:05:43 (1786691143) [ 2138.688707] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 03:05:48 (1786691148) [ 2143.564669] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 03:05:53 (1786691153) [ 2149.213143] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 03:05:59 (1786691159) [ 2155.461419] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 03:06:05 (1786691165) [ 2162.846413] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 03:06:12 (1786691172) [ 2169.331564] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 03:06:19 (1786691179) [ 2175.353798] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 03:06:25 (1786691185) [ 2181.100736] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 03:06:31 (1786691191) [ 2186.646765] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 03:06:36 (1786691196) [ 2191.520501] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 03:06:41 (1786691201) [ 2196.379248] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 03:06:46 (1786691206) [ 2201.772286] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 03:06:52 (1786691212) [ 2206.861889] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 03:06:57 (1786691217) [ 2213.115540] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 03:07:03 (1786691223) [ 2219.577857] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 03:07:09 (1786691229) [ 2219.701601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.728129] LNet: 136654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.734796] LNetError: 136654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.771139] LNet: Removed LNI 192.168.206.47@tcp2 [ 2219.795418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.909720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.934509] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2221.136235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.152815] LNet: 136799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.165860] LNetError: 136799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.181361] LNet: Removed LNI 192.168.206.47@tcp [ 2221.197457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.245108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2221.266541] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2227.702977] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 03:07:17 (1786691237) [ 2234.370875] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 03:07:24 (1786691244) [ 2240.441836] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 03:07:30 (1786691250) [ 2247.567351] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 03:07:37 (1786691257) [ 2253.436823] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 03:07:43 (1786691263) [ 2259.635839] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 03:07:49 (1786691269) [ 2266.426775] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 03:07:56 (1786691276) [ 2273.692674] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 03:08:03 (1786691283) [ 2280.602898] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 03:08:10 (1786691290) [ 2286.521986] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:08:16 (1786691296) [ 2291.671691] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:08:21 (1786691301) [ 2296.312787] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:08:26 (1786691306) [ 2301.179920] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2051 sec ========== 03:08:31 (1786691311) [ 2301.373523] LNet: 144261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.380579] LNetError: 144261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.392845] LNet: Removed LNI 192.168.206.47@tcp [ 2301.924921] Key type .llcrypt unregistered [ 2301.928987] Key type ._llcrypt unregistered [ 2312.362359] Key type ._llcrypt registered [ 2312.367572] Key type .llcrypt registered [ 2312.640547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2312.652442] alg: No test for adler32 (adler32-zlib) [ 2313.683863] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 2313.877609] LNet: Added LNI 192.168.206.47@tcp [8/256/0/180] [ 2315.543509] Key type lgssc registered [ 2316.568379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.357217] Lustre: Mounted lustre-client - version 2.17.57_1_g9a8e296 [ 2382.291945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2391.117519] Lustre: Unmounted lustre-client [ 2413.371016] Key type lgssc unregistered [ 2413.638703] LNet: 147219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.650561] LNetError: 147219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.668957] LNet: Removed LNI 192.168.206.47@tcp [ 2414.592586] Key type .llcrypt unregistered [ 2414.596842] Key type ._llcrypt unregistered