[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485956223 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002339] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006901] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009114] LSM: Security Framework initializing [ 0.010036] Yama: becoming mindful. [ 0.011023] SELinux: Initializing. [ 0.011614] *** VALIDATE selinux *** [ 0.019431] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024550] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027085] *** VALIDATE tmpfs *** [ 0.028331] *** VALIDATE proc *** [ 0.029170] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035022] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.039363] debug: unmapping init [mem 0xffffffff98059000-0xffffffff98060fff] [ 0.041870] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042557] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.043868] ... bit width: 48 [ 0.044014] ... generic registers: 4 [ 0.045012] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047015] ... fixed-purpose events: 3 [ 0.048010] ... event mask: 000000070000000f [ 0.049268] rcu: Hierarchical SRCU implementation. [ 0.051443] smp: Bringing up secondary CPUs ... [ 0.052582] x86: Booting SMP configuration: [ 0.053018] .... node #0, CPUs: #1 #2 #3 [ 0.056082] smp: Brought up 1 node, 4 CPUs [ 0.057815] smpboot: Max logical packages: 1 [ 0.058007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145682] node 0 deferred pages initialised in 83ms [ 0.150008] devtmpfs: initialized [ 0.151300] x86/mm: Memory block size: 128MB [ 0.154783] gcov: version magic: 0x41383552 [ 0.157317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163319] pinctrl core: initialized pinctrl subsystem [ 0.165188] [ 0.165729] ************************************************************* [ 0.168012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171011] ** ** [ 0.173012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175018] ** ** [ 0.177014] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.182013] ** your system. ** [ 0.184013] ** ** [ 0.186013] ** If you see this message and you are not debugging the ** [ 0.188016] ** kernel, report this immediately to your vendor! ** [ 0.190016] ** ** [ 0.192012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195013] ************************************************************* [ 0.197906] NET: Registered protocol family 16 [ 0.199679] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209009] cpuidle: using governor menu [ 0.211247] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212309] PCI: Using configuration type 1 for base access [ 0.214121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224048] cryptd: max_cpu_qlen set to 1000 [ 0.228376] ACPI: Added _OSI(Module Device) [ 0.229019] ACPI: Added _OSI(Processor Device) [ 0.231016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237260] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243011] ACPI: Interpreter enabled [ 0.245063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246016] ACPI: Using IOAPIC for interrupt routing [ 0.248124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261851] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274441] acpiphp: Slot [2] registered [ 0.276086] acpiphp: Slot [5] registered [ 0.277130] acpiphp: Slot [6] registered [ 0.279208] acpiphp: Slot [3] registered [ 0.281111] acpiphp: Slot [4] registered [ 0.282106] acpiphp: Slot [7] registered [ 0.283155] acpiphp: Slot [8] registered [ 0.285106] acpiphp: Slot [9] registered [ 0.286149] acpiphp: Slot [10] registered [ 0.288104] acpiphp: Slot [11] registered [ 0.290154] acpiphp: Slot [12] registered [ 0.291097] acpiphp: Slot [13] registered [ 0.292000] acpiphp: Slot [14] registered [ 0.292000] acpiphp: Slot [15] registered [ 0.294111] acpiphp: Slot [16] registered [ 0.296116] acpiphp: Slot [17] registered [ 0.297121] acpiphp: Slot [18] registered [ 0.299098] acpiphp: Slot [19] registered [ 0.300099] acpiphp: Slot [20] registered [ 0.302105] acpiphp: Slot [21] registered [ 0.303101] acpiphp: Slot [22] registered [ 0.305108] acpiphp: Slot [23] registered [ 0.307137] acpiphp: Slot [24] registered [ 0.308092] acpiphp: Slot [25] registered [ 0.309096] acpiphp: Slot [26] registered [ 0.310092] acpiphp: Slot [27] registered [ 0.311096] acpiphp: Slot [28] registered [ 0.312101] acpiphp: Slot [29] registered [ 0.313133] acpiphp: Slot [30] registered [ 0.315153] acpiphp: Slot [31] registered [ 0.316000] PCI host bridge to bus 0000:00 [ 0.316000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321337] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322332] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327010] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.329962] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338201] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339720] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.345063] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.359014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.364014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.370605] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.375015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.388012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395391] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.404016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.412014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.420167] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.423382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.426348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.430398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.433244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.438202] iommu: Default domain type: Passthrough [ 0.441469] SCSI subsystem initialized [ 0.443131] ACPI: bus type USB registered [ 0.444112] usbcore: registered new interface driver usbfs [ 0.446069] usbcore: registered new interface driver hub [ 0.447077] usbcore: registered new device driver usb [ 0.449175] pps_core: LinuxPPS API ver. 1 registered [ 0.451011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.454059] PTP clock support registered [ 0.456389] EDAC MC: Ver: 3.0.0 [ 0.458450] PCI: Using ACPI for IRQ routing [ 0.460810] NetLabel: Initializing [ 0.463010] NetLabel: domain hash size = 128 [ 0.464011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.466088] NetLabel: unlabeled traffic allowed by default [ 0.469090] vgaarb: loaded [ 0.470233] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.471016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.477152] clocksource: Switched to clocksource kvm-clock [ 0.586462] VFS: Disk quotas dquot_6.6.0 [ 0.588039] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.590333] *** VALIDATE ramfs *** [ 0.591448] *** VALIDATE hugetlbfs *** [ 0.592833] pnp: PnP ACPI init [ 0.595911] pnp: PnP ACPI: found 6 devices [ 0.612860] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.615470] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.616769] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.618243] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.619496] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.620803] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.622469] NET: Registered protocol family 2 [ 0.624351] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.629112] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.632556] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.636438] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.640919] TCP: Hash tables configured (established 65536 bind 65536) [ 0.643160] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.646158] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.648693] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.651595] NET: Registered protocol family 1 [ 0.654219] RPC: Registered named UNIX socket transport module. [ 0.655728] RPC: Registered udp transport module. [ 0.656792] RPC: Registered tcp transport module. [ 0.658144] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.659898] NET: Registered protocol family 44 [ 0.661548] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.663346] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.665066] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.666967] PCI: CLS 0 bytes, default 64 [ 0.668367] Unpacking initramfs... [ 2.083850] debug: unmapping init [mem 0xffff9a0cbcc64000-0xffff9a0cbffcffff] [ 2.089038] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.091416] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.095473] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.585267] Initialise system trusted keyrings [ 2.586770] Key type blacklist registered [ 2.588467] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.597337] zbud: loaded [ 2.600108] *** VALIDATE nfs *** [ 2.601416] *** VALIDATE nfs4 *** [ 2.602707] pstore: using deflate compression [ 2.605206] Platform Keyring initialized [ 2.710215] NET: Registered protocol family 38 [ 2.711823] Key type asymmetric registered [ 2.713859] Asymmetric key parser 'x509' registered [ 2.715800] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.718128] io scheduler mq-deadline registered [ 2.719608] io scheduler kyber registered [ 2.721375] io scheduler bfq registered [ 2.723121] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.725898] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.728659] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.731489] ACPI: Power Button [PWRF] [ 2.736959] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.742141] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.748904] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.773258] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.797832] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.800846] Non-volatile memory driver v1.3 [ 2.801717] Linux agpgart interface v0.103 [ 2.825451] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 2.827710] vda: detected capacity change from 0 to 77111296 [ 2.840049] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.842698] vdb: detected capacity change from 0 to 1073741824 [ 2.846927] libphy: Fixed MDIO Bus: probed [ 2.850744] usbcore: registered new interface driver usbserial_generic [ 2.852124] usbserial: USB Serial support registered for generic [ 2.853463] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.857148] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.858177] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.859732] mousedev: PS/2 mouse device common for all mice [ 2.863514] rtc_cmos 00:05: RTC can wake from S4 [ 2.864645] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.867599] rtc_cmos 00:05: registered as rtc0 [ 2.868526] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.869450] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.872885] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.873315] intel_pstate: CPU model not supported [ 2.877779] hid: raw HID events driver (C) Jiri Kosina [ 2.879889] usbcore: registered new interface driver usbhid [ 2.881988] usbhid: USB HID core driver [ 2.883677] drop_monitor: Initializing network drop monitor service [ 2.886102] Initializing XFRM netlink socket [ 2.888099] NET: Registered protocol family 10 [ 2.890662] Segment Routing with IPv6 [ 2.891929] NET: Registered protocol family 17 [ 2.893882] mpls_gso: MPLS GSO support [ 2.900463] RAS: Correctable Errors collector initialized. [ 2.902251] AVX version of gcm_enc/dec engaged. [ 2.903539] AES CTR mode by8 optimization enabled [ 2.970590] sched_clock: Marking stable (2970559885, 0)->(3862856227, -892296342) [ 2.973721] registered taskstats version 1 [ 2.976282] Loading compiled-in X.509 certificates [ 2.977679] zswap: loaded using pool lzo/zbud [ 3.006053] Key type big_key registered [ 3.016032] Key type encrypted registered [ 3.016976] ima: No TPM chip found, activating TPM-bypass! [ 3.018699] ima: Allocated hash algorithm: sha1 [ 3.019677] ima: No architecture policies found [ 3.020811] evm: Initialising EVM extended attributes: [ 3.022423] evm: security.selinux [ 3.023572] evm: security.ima [ 3.024731] evm: security.capability [ 3.026180] evm: HMAC attrs: 0x1 [ 3.028501] rtc_cmos 00:05: setting system clock to 2026-10-01 23:33:37 UTC (1790897617) [ 3.034450] debug: unmapping init [mem 0xffffffff99003000-0xffffffff991fffff] [ 3.036427] debug: unmapping init [mem 0xffffffff97d82000-0xffffffff98058fff] [ 3.045097] Write protecting the kernel read-only data: 28672k [ 3.047638] debug: unmapping init [mem 0xffffffff96403000-0xffffffff965fffff] [ 3.049320] debug: unmapping init [mem 0xffffffff96d14000-0xffffffff96dfffff] [ 3.077757] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.085693] systemd[1]: Detected virtualization kvm. [ 3.087341] systemd[1]: Detected architecture x86-64. [ 3.089173] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.119775] systemd[1]: No hostname configured. [ 3.121155] systemd[1]: Set hostname to . [ 3.122331] random: systemd: uninitialized urandom read (16 bytes read) [ 3.123670] systemd[1]: Initializing machine ID from random generator. [ 3.158809] random: ln: uninitialized urandom read (6 bytes read) [ 3.229666] random: systemd: uninitialized urandom read (16 bytes read) [ 3.232772] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.242485] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.259741] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.872802] device-mapper: uevent: version 1.0.3 [ 3.874658] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.501553] virtio_net virtio0 ens2: renamed from eth0 [ 4.584039] scsi host0: ata_piix [ 4.604479] scsi host1: ata_piix [ 4.606030] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.607846] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.163931] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.543065] random: crng init done [ 9.544540] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.843074] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.998988] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.248237] SELinux: Disabled at runtime. [ 11.310577] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.320902] systemd[1]: Detected virtualization kvm. [ 11.323487] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.929512] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.932867] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.939736] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.944507] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.948524] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.956565] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.967427] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.108449] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.466765] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 12.816769] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.845169] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.977273] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.992781] EDAC sbridge: Ver: 1.1.2 [ 14.237300] Key type dns_resolver registered [ 14.547928] NFS: Registering the id_resolver key type [ 14.550088] Key type id_resolver registered [ 14.551676] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-client login: [ 52.029673] libcfs: loading out-of-tree module taints kernel. [ 52.294198] Key type ._llcrypt registered [ 52.308671] Key type .llcrypt registered [ 53.354209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 53.373756] alg: No test for adler32 (adler32-zlib) [ 55.270739] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 56.481331] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 58.255890] Key type lgssc registered [ 60.627794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 63.405818] hrtimer: interrupt took 30647959 ns [ 218.388745] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d [ 224.602973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.863497] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing check_logdir /tmp/testlogs/ [ 244.191183] Lustre: lustre-OST0000-osc-ffff9a0d052df000: disconnect after 23s idle [ 244.983569] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing yml_node [ 248.710517] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 250.971733] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 253.263366] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 255.125608] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Oct 1 19:37:48 EDT 2026 [ 271.494103] Lustre: Unmounted lustre-client [ 303.062568] Key type lgssc unregistered [ 303.473660] LNet: 8401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.486542] LNetError: 8401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 303.517497] LNet: Removed LNI 192.168.202.37@tcp [ 304.764100] Key type .llcrypt unregistered [ 304.766972] Key type ._llcrypt unregistered [ 340.331789] Key type ._llcrypt registered [ 340.334217] Key type .llcrypt registered [ 340.714581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 340.770266] alg: No test for adler32 (adler32-zlib) [ 341.745953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 341.804260] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 341.821486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 346.494504] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 346.675753] LNet: 9288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.690764] LNetError: 9288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.727283] LNet: Removed LNI 192.168.202.37@tcp [ 347.245981] Key type .llcrypt unregistered [ 347.248555] Key type ._llcrypt unregistered [ 353.336840] Key type ._llcrypt registered [ 353.341682] Key type .llcrypt registered [ 354.784820] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:39:27 (1790897967) [ 355.186982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.198902] alg: No test for adler32 (adler32-zlib) [ 356.130777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.201595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.218668] LNet: 9706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.263391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.352527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.377981] LNet: 9713:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 363.373853] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:39:36 (1790897976) [ 363.542470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.555615] LNet: 10292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.570373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.615821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 370.607623] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:39:43 (1790897983) [ 370.810964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.830163] LNet: 10877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.851440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.964493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 377.742970] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:39:50 (1790897990) [ 378.103396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.119076] LNet: 11461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.137870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.233234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 385.222976] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:39:58 (1790897998) [ 385.287708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.312649] LNet: 12042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.323933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.370562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 385.721758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 392.597582] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:40:05 (1790898005) [ 393.043299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.094299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.296404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 394.007763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 394.637410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 402.554905] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:40:15 (1790898015) [ 402.660698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.683803] LNet: 13227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.691401] LNet: 13227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 402.709561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.815887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 411.046818] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:40:23 (1790898023) [ 411.455734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.565736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.747289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 411.803632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 412.745759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 412.792603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 413.539671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 413.612250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 414.660652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 414.738995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 415.501322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 415.586685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 416.385404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 416.441749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.113914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 417.148198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 417.637823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 417.696811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 418.375306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 418.440265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 419.083294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 419.128958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 420.033264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 420.118746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 420.511479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 420.572777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.863865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 420.972477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 431.384214] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:40:43 (1790898043) [ 431.474488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.543222] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.550876] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 431.623090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.712266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 431.766577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 438.191763] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:40:51 (1790898051) [ 438.237173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.258081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.354585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 438.409043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 446.988351] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:40:59 (1790898059) [ 447.123349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.194311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.335848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 447.373228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 454.436178] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:41:07 (1790898067) [ 454.565900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.622256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.748734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 454.789892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 462.366521] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:41:15 (1790898075) [ 462.507183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.537477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.589100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 462.633956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 468.840897] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:41:21 (1790898081) [ 469.015262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.042082] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.051916] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 469.105618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.191704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 469.245434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 475.988956] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:41:29 (1790898089) [ 476.257857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.305720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.401223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.836076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.786985] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:41:36 (1790898096) [ 484.180187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.255873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.477133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.829899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.756860] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:41:46 (1790898106) [ 493.861777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.924167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.076708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.357716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.242208] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:41:55 (1790898115) [ 502.411964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.509806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.141944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.529801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.780519] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:42:03 (1790898123) [ 509.885633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.955876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.023531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.048000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.475777] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:42:09 (1790898129) [ 516.615881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.671125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.822730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.863795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.817760] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:42:17 (1790898137) [ 523.959667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.013388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.119904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 530.007699] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:42:23 (1790898143) [ 530.121503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.161946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.243451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 536.601097] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:42:29 (1790898149) [ 536.680815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.704960] LNet: 22865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.707980] LNet: 22865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 536.730521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.814778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 537.284035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 545.205419] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:42:37 (1790898157) [ 545.287560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.359577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.479798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 545.508774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 552.966751] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:42:45 (1790898165) [ 553.091076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.184647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.263621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 553.294968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 560.484839] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:42:53 (1790898173) [ 560.623784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.682458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.825637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 560.870940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 568.200564] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:43:01 (1790898181) [ 568.320620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.405014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.557719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 568.647776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 576.409677] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:43:08 (1790898188) [ 576.510743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.591314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.650810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 576.720501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 576.735934] LNetError: 25819:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 576.766602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 576.797203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 583.179310] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:43:16 (1790898196) [ 583.336071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.395908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.448607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wIiOuP2YcJ [ 583.494402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wIiOuP2YcJ [ 583.569203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wIiOuP2YcJ [ 589.512765] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:43:22 (1790898202) [ 589.729321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.746097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.767895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 589.788662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 589.817679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 595.701482] Lustre: DEBUG MARKER: == sanity-lnet test 29: peer show: unknown peer should be reported ========================================================== 19:43:28 (1790898208) [ 595.775789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.820163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.131468] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:43:35 (1790898215) [ 603.204833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.248798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.335747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.376523] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 605.087416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 606.830374] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 608.767846] Key type lgssc registered [ 610.857243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 616.951399] LNetError: 28156:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.37@tcp99 has no available nets [ 620.205250] Key type lgssc unregistered [ 620.736964] LNetError: 29276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.778642] LNet: Removed LNI 192.168.202.37@tcp [ 621.884562] Key type .llcrypt unregistered [ 621.886143] Key type ._llcrypt unregistered [ 630.963828] Key type ._llcrypt registered [ 630.966216] Key type .llcrypt registered [ 632.405850] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:44:05 (1790898245) [ 633.536443] Key type .llcrypt unregistered [ 633.539312] Key type ._llcrypt unregistered [ 640.379348] Key type ._llcrypt registered [ 640.384750] Key type .llcrypt registered [ 640.509138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.517109] alg: No test for adler32 (adler32-zlib) [ 641.393512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.432993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 641.444259] LNet: 30198:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 641.448278] LNet: 30198:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 641.451477] LNet: 30198:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 641.459000] LNet: Message forwarding will be enabled in 2s [ 644.511182] LNet: Message forwarding enabled [ 644.890157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.922209] LNet: Message forwarding will stop in 2s to 6s [ 648.620189] LNet: Message forwarding disabled [ 649.707779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 649.731501] LNet: 30296:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 649.742138] LNet: 30296:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 649.750144] LNet: 30296:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 649.763598] LNet: Message forwarding will be enabled in 2s [ 649.952758] LNet: 30309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.737234] Key type .llcrypt unregistered [ 650.746203] Key type ._llcrypt unregistered [ 664.298849] Key type ._llcrypt registered [ 664.303298] Key type .llcrypt registered [ 665.803669] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:44:38 (1790898278) [ 666.143158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.160471] alg: No test for adler32 (adler32-zlib) [ 667.110296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.181739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 673.278900] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:44:46 (1790898286) [ 673.467393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.495806] LNet: 31712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.512538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.542968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 673.568188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 673.606623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 673.705383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 673.755358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 673.787766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 673.832618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 673.885516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 673.921496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 673.952912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 674.020755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 674.089757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 674.156278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 674.227625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 674.282473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 674.337983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 674.380541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 674.414962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 674.473628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 674.521184] LNetError: 31755:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 682.589394] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:44:54 (1790898294) [ 682.777092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.807475] LNet: 32322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.859950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.029865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.207232] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:45:04 (1790898304) [ 692.689414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg237-server [ 694.271242] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:45:07 (1790898307) [ 695.128681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.157991] LNet: 33257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.209062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.281690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.332124] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 696.704106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[38-42]@tcp [ 696.730305] LNet: 33390:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 696.791981] LNetError: 33261:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.38@tcp is being used as a gateway but routing feature is not turned on [ 696.804784] LNetError: 33261:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.38@tcp has gone from up to down [ 699.810453] LNetError: 33265:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.39@tcp has gone from up to down [ 699.852116] LNetError: 33265:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 707.284761] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:45:19 (1790898319) [ 709.710083] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg237-server [ 711.474623] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:45:24 (1790898324) [ 711.785186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.808168] LNet: 34327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.813837] LNetError: 34327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.870541] LNet: Removed LNI 192.168.202.37@tcp [ 712.940896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.041638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.070931] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 713.915531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[38-43/2]@tcp [ 713.970222] LNet: 34445:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 713.980434] LNet: 34445:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 714.013575] LNetError: 34332:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.38@tcp is being used as a gateway but routing feature is not turned on [ 714.019726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[38-43/2]@tcp [ 714.036215] LNetError: 34332:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 714.062096] LNetError: 34332:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.38@tcp has gone from up to down [ 721.557334] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:45:34 (1790898334) [ 721.679906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.708811] LNet: 35037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.724960] LNetError: 35037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.747943] LNet: Removed LNI 192.168.202.37@tcp [ 721.766606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.836605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 721.890915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.919564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.968282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 722.030603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.063500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.156515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 722.278416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.314992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.395681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 722.487296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.519188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.582698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 722.693741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.749729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.855374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.028990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.074966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.147832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 723.228478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.255978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.293847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.360606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.387634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.428285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 723.504457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.547828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.590553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.663858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.703597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.738561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 723.782735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.816742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.862932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.587091] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:45:43 (1790898343) [ 732.539097] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg237-server [ 734.662058] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:45:47 (1790898347) [ 736.720970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg237-server [ 738.351938] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:45:51 (1790898351) [ 738.677172] LNet: 36441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.691331] LNet: 36441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 739.363349] Key type .llcrypt unregistered [ 739.367612] Key type ._llcrypt unregistered [ 745.511945] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 745.760917] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 745.776881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 745.787405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 748.536248] Key type ._llcrypt registered [ 748.538832] Key type .llcrypt registered [ 748.683243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 748.697132] alg: No test for adler32 (adler32-zlib) [ 749.727841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.798790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 749.821083] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 750.547356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.567058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 752.794936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 752.813577] LNet: Removed LNI 10.1.2.3@tcp [ 752.925730] LNet: 37161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.930105] LNetError: 37161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.940727] LNet: Removed LNI 192.168.202.37@tcp [ 753.737703] Key type .llcrypt unregistered [ 753.739881] Key type ._llcrypt unregistered [ 766.319164] Key type ._llcrypt registered [ 766.321896] Key type .llcrypt registered [ 767.928785] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:46:20 (1790898380) [ 768.580696] Key type .llcrypt unregistered [ 768.587439] Key type ._llcrypt unregistered [ 774.728396] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.964309] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.971363] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.979924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 777.461436] Key type ._llcrypt registered [ 777.464338] Key type .llcrypt registered [ 777.596340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.606751] alg: No test for adler32 (adler32-zlib) [ 778.575623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.633318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.663680] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 779.429719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 779.453237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.492885] LNet: 38655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.506531] LNetError: 38655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 782.509425] LNet: Removed LNI 10.1.2.3@tcp [ 782.522539] LNet: Removed LNI 192.168.202.37@tcp [ 783.046621] Key type .llcrypt unregistered [ 783.055221] Key type ._llcrypt unregistered [ 794.866545] Key type ._llcrypt registered [ 794.868868] Key type .llcrypt registered [ 796.145161] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:46:49 (1790898409) [ 796.577350] Key type .llcrypt unregistered [ 796.581431] Key type ._llcrypt unregistered [ 802.906265] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.116594] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.120577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.123939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.712832] Key type ._llcrypt registered [ 805.717784] Key type .llcrypt registered [ 806.085868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.118289] alg: No test for adler32 (adler32-zlib) [ 807.154119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.270172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 807.306599] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 809.341649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 809.358992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.432406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 811.455773] LNet: Removed LNI 10.1.2.3@tcp [ 811.476367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 811.513837] LNet: Removed LNI 10.1.2.31@tcp [ 811.526776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.553806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.332422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 812.357652] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 813.375985] LNet: 40572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.383335] LNetError: 40572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 813.388283] LNet: Removed LNI 10.1.2.31@tcp [ 813.412269] LNet: Removed LNI 10.1.2.3@tcp [ 813.968534] Key type .llcrypt unregistered [ 813.970647] Key type ._llcrypt unregistered [ 826.241190] Key type ._llcrypt registered [ 826.245172] Key type .llcrypt registered [ 827.866648] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:47:20 (1790898440) [ 828.310970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 828.324798] alg: No test for adler32 (adler32-zlib) [ 829.266336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.306731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.329959] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 830.146298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.166221] LNet: 41525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.172277] LNetError: 41525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.187759] LNet: Removed LNI 192.168.202.37@tcp [ 830.233708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.314842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.37@tcp [ 830.350056] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 831.252375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.37@tcp [ 831.279617] LNet: Removed LNI 192.168.202.37@tcp [ 831.339429] LNet: 41660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.012746] Key type .llcrypt unregistered [ 832.016175] Key type ._llcrypt unregistered [ 843.139398] Key type ._llcrypt registered [ 843.141620] Key type .llcrypt registered [ 844.571957] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:47:37 (1790898457) [ 846.410671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg237-server [ 847.985989] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:47:41 (1790898461) [ 848.462525] Key type .llcrypt unregistered [ 848.464856] Key type ._llcrypt unregistered [ 854.470578] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 854.671827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 854.678206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 854.684295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.191703] Key type ._llcrypt registered [ 857.194657] Key type .llcrypt registered [ 857.405893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.421774] alg: No test for adler32 (adler32-zlib) [ 858.336907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.405363] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 859.157356] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 860.883458] LNet: 43495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.887916] LNetError: 43495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.891960] LNet: Removed LNI 192.168.202.37@tcp1 [ 860.904178] LNet: Removed LNI 10.1.2.3@tcp2 [ 861.456402] Key type .llcrypt unregistered [ 861.458457] Key type ._llcrypt unregistered [ 872.376772] Key type ._llcrypt registered [ 872.378465] Key type .llcrypt registered [ 873.512338] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:48:06 (1790898486) [ 873.860533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.872381] alg: No test for adler32 (adler32-zlib) [ 874.773857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.834366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.878932] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 875.718696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.738515] LNet: 44463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.743981] LNetError: 44463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.762213] LNet: Removed LNI 192.168.202.37@tcp [ 875.773691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.829449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 875.842663] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 882.279986] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:48:15 (1790898495) [ 882.464983] LNet: 45055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.469449] LNetError: 45055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.494359] LNet: Removed LNI 192.168.202.37@tcp [ 883.311644] Key type .llcrypt unregistered [ 883.317099] Key type ._llcrypt unregistered [ 890.209647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.363022] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.382721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.389259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 893.046293] Key type ._llcrypt registered [ 893.053186] Key type .llcrypt registered [ 893.303337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.348229] alg: No test for adler32 (adler32-zlib) [ 894.277630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.348964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.394166] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 895.190726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 895.209478] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.233208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.284962] LNet: 45796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.298214] LNetError: 45796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.310291] LNet: Removed LNI 10.1.2.3@tcp [ 897.348810] LNet: Removed LNI 192.168.202.37@tcp [ 897.385778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.439580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 897.458487] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 897.462544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.888286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.911415] LNet: 45823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.921890] LNetError: 45823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.927964] LNetError: 45823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 897.934451] LNet: Removed LNI 10.1.2.3@tcp [ 897.954168] LNet: Removed LNI 192.168.202.37@tcp [ 897.984840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.094286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 898.114727] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 898.117656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.743689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 898.786331] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 899.646537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.667891] LNet: 45985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.671797] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.678380] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 899.683075] LNet: Removed LNI 10.1.2.3@tcp [ 899.684523] LNet: Removed LNI 192.168.202.37@tcp [ 899.698389] LNet: Removed LNI 192.168.202.37@tcp2 [ 899.715961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.785274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 899.813873] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 899.816980] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.827042] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 906.698841] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:48:40 (1790898520) [ 906.880285] LNet: 46584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.885499] LNetError: 46584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.890183] LNetError: 46584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 906.896324] LNet: Removed LNI 10.1.2.3@tcp [ 906.898703] LNet: Removed LNI 192.168.202.37@tcp [ 906.911146] LNet: Removed LNI 192.168.202.37@tcp2 [ 907.526771] Key type .llcrypt unregistered [ 907.531280] Key type ._llcrypt unregistered [ 912.954218] Key type ._llcrypt registered [ 912.957813] Key type .llcrypt registered [ 913.171091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.187560] alg: No test for adler32 (adler32-zlib) [ 914.043422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.111926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.133753] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 914.710252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.719608] LNet: 46978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.725988] LNetError: 46978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.739825] LNet: Removed LNI 192.168.202.37@tcp [ 914.770511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.833363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 914.858066] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 915.311844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.326566] LNet: 47008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.331699] LNetError: 47008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.342144] LNet: Removed LNI 192.168.202.37@tcp [ 915.355797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.395623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 915.415401] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 915.709505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.732930] LNet: Removed LNI 192.168.202.37@tcp [ 915.757932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.838626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 915.885839] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 916.130709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.314141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 916.319390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 916.329563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.631103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.656666] LNet: 47113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.661825] LNet: 47113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 918.668292] LNetError: 47113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.672088] LNetError: 47113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 918.694141] LNet: Removed LNI 192.168.202.37@tcp [ 918.708965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.797219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 918.831236] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 923.986706] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:48:57 (1790898537) [ 924.074654] LNet: 47702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.086196] LNetError: 47702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.111745] LNet: Removed LNI 192.168.202.37@tcp [ 924.624179] Key type .llcrypt unregistered [ 924.628531] Key type ._llcrypt unregistered [ 930.630768] Key type ._llcrypt registered [ 930.637434] Key type .llcrypt registered [ 930.828596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.862211] alg: No test for adler32 (adler32-zlib) [ 931.780107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.859983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 931.895819] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 932.827756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 932.847645] LNet: 48107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.852805] LNetError: 48107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.870594] LNet: Removed LNI 192.168.202.37@tcp [ 932.891351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.968692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 932.998172] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 939.717433] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:49:12 (1790898552) [ 939.791204] LNet: 48691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.798485] LNetError: 48691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.822889] LNet: Removed LNI 192.168.202.37@tcp [ 940.389577] Key type .llcrypt unregistered [ 940.395395] Key type ._llcrypt unregistered [ 946.771668] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 946.936781] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 946.939419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 946.950877] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.586988] Key type ._llcrypt registered [ 949.589749] Key type .llcrypt registered [ 949.914790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.949619] alg: No test for adler32 (adler32-zlib) [ 950.950363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.001894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.015452] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 951.901977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.932244] LNet: 49148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.946547] LNetError: 49148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.964885] LNet: Removed LNI 192.168.202.37@tcp [ 952.009927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.102722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 952.131403] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 959.134251] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:49:32 (1790898572) [ 959.262336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.297977] LNet: 49742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.317481] LNetError: 49742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.336048] LNet: Removed LNI 192.168.202.37@tcp [ 959.405892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.595473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 965.493270] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:49:38 (1790898578) [ 965.718517] LNet: 50329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.413621] Key type .llcrypt unregistered [ 966.417235] Key type ._llcrypt unregistered [ 972.466985] Key type ._llcrypt registered [ 972.469449] Key type .llcrypt registered [ 972.708520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.734387] alg: No test for adler32 (adler32-zlib) [ 973.649640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.679576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.697204] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 974.541396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.556668] LNet: 50725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.561739] LNetError: 50725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.575148] LNet: Removed LNI 192.168.202.37@tcp [ 974.592664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.639428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 974.660413] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 980.329847] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:49:53 (1790898593) [ 980.466713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.482172] LNet: 51319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.504556] LNetError: 51319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.517219] LNet: Removed LNI 192.168.202.37@tcp [ 980.537846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.603962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 986.547290] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:49:59 (1790898599) [ 988.218735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 990.033988] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:50:03 (1790898603) [ 991.444676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 993.649312] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:50:06 (1790898606) [ 995.107888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 996.704266] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:50:09 (1790898609) [ 998.038526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1000.015628] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:50:13 (1790898613) [ 1001.735539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1003.405953] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:50:16 (1790898616) [ 1003.712943] LNet: 53341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.456275] Key type .llcrypt unregistered [ 1004.458733] Key type ._llcrypt unregistered [ 1010.399548] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.614318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.620810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.629428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1013.451672] Key type ._llcrypt registered [ 1013.454865] Key type .llcrypt registered [ 1013.724434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1013.760494] alg: No test for adler32 (adler32-zlib) [ 1014.762967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.870094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1014.901252] LNet: Added LNI 192.168.202.37@tcp [8/512/0/180] [ 1014.908238] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1014.912545] LNet: 53664:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1015.049329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.113309] LNet: 53683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.148741] LNetError: 53683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.170042] LNet: Removed LNI 10.1.2.3@tcp [ 1015.177038] LNetError: 53683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.209569] LNet: Removed LNI 192.168.202.37@tcp [ 1015.328527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.547337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.580639] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1015.700347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.732219] LNet: 53707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.747945] LNetError: 53707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.795807] LNet: Removed LNI 192.168.202.37@tcp [ 1015.826519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.916425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1015.945245] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1015.948926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.951658] LNet: 53715:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1016.098889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.145232] LNet: 53733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.153793] LNetError: 53733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.163730] LNet: Removed LNI 10.1.2.3@tcp [ 1016.167439] LNetError: 53733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.191060] LNet: Removed LNI 192.168.202.37@tcp [ 1016.253364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.391167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1016.466152] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1016.479122] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.490274] LNet: 53741:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1016.729888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.745938] LNet: 53767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.750823] LNetError: 53767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.756144] LNet: Removed LNI 10.1.2.3@tcp [ 1016.758512] LNetError: 53767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.773985] LNet: Removed LNI 192.168.202.37@tcp [ 1016.804601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.879357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1016.906426] LNet: Added LNI 192.168.202.37@tcp [8/256/0/182] [ 1016.913263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1017.076871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1017.097845] LNet: 53795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.103871] LNetError: 53795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1017.110364] LNet: Removed LNI 10.1.2.3@tcp [ 1017.112967] LNetError: 53795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.125983] LNet: Removed LNI 192.168.202.37@tcp [ 1017.146553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.265213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1017.303411] LNet: Added LNI 192.168.202.37@tcp [8/256/0/182] [ 1017.312131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1017.319682] LNet: 53803:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1017.420899] LNet: Removed LNI 10.1.2.3@tcp [ 1017.446321] LNet: Removed LNI 192.168.202.37@tcp [ 1018.061762] Key type .llcrypt unregistered [ 1018.062855] Key type ._llcrypt unregistered [ 1035.135479] Key type ._llcrypt registered [ 1035.139659] Key type .llcrypt registered [ 1036.292358] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:50:49 (1790898649) [ 1037.965443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1040.126992] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:50:52 (1790898652) [ 1040.948244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.975148] alg: No test for adler32 (adler32-zlib) [ 1042.007396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.122965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1042.159972] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1043.044827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.37 [ 1043.068103] LNet: 55069:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1043.075099] LNet: Removed LNI 192.168.202.37@tcp [ 1049.518348] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:51:02 (1790898662) [ 1049.879319] LNet: 55640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.888372] LNetError: 55640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.926844] LNet: Removed LNI 192.168.202.37@tcp [ 1050.454487] Key type .llcrypt unregistered [ 1050.457820] Key type ._llcrypt unregistered [ 1056.508837] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.899342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1059.902880] Key type ._llcrypt registered [ 1059.905388] Key type .llcrypt registered [ 1060.041197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.052733] alg: No test for adler32 (adler32-zlib) [ 1061.334036] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1069.238318] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:51:22 (1790898682) [ 1069.565513] LNet: 56553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.570569] LNetError: 56553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.584575] LNet: Removed LNI 10.1.2.3@tcp [ 1070.310849] Key type .llcrypt unregistered [ 1070.313710] Key type ._llcrypt unregistered [ 1076.116239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1076.434119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1076.445342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1076.454714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.409138] Key type ._llcrypt registered [ 1079.410921] Key type .llcrypt registered [ 1079.737050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.788491] alg: No test for adler32 (adler32-zlib) [ 1081.008346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1088.702934] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:51:41 (1790898701) [ 1090.562678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1092.335954] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:51:45 (1790898705) [ 1092.504653] LNet: 57732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.521520] LNetError: 57732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1092.566587] LNet: Removed LNI 10.1.2.3@tcp [ 1093.337577] Key type .llcrypt unregistered [ 1093.344384] Key type ._llcrypt unregistered [ 1102.046701] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1102.382845] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1102.385703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1102.392936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1105.056817] Key type ._llcrypt registered [ 1105.058483] Key type .llcrypt registered [ 1105.199916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1105.221639] alg: No test for adler32 (adler32-zlib) [ 1106.161777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.291628] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1114.836871] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:52:07 (1790898727) [ 1115.384559] LNet: 58965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.391856] LNetError: 58965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1115.404449] LNet: Removed LNI 10.1.2.3@tcp [ 1115.965827] Key type .llcrypt unregistered [ 1115.969428] Key type ._llcrypt unregistered [ 1123.292594] Key type ._llcrypt registered [ 1123.304178] Key type .llcrypt registered [ 1123.685782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.710294] alg: No test for adler32 (adler32-zlib) [ 1124.689496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.741637] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1132.228810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1133.070317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_interrupt -r 1 [ 1133.127281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_interrupt -r 1 [ 1133.174652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1133.204947] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1133.496938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.546942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.573615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.600682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.620918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1133.657906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.714661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.742317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.783593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.804855] LNet: 59403:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.819822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.866655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.908770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.942266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.979226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.983211] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp [ 1134.018243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.051566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.617336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_dropped -r 1 [ 1134.659417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_dropped -r 1 [ 1134.710187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1134.734934] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1134.985286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.015731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.039301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.061312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.089879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1135.117531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.143436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.171188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.202406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.226306] LNet: 59515:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.255662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.286850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.323982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.396740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.437389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.473677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.503666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.066265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_aborted -r 1 [ 1136.100830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_aborted -r 1 [ 1136.141385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1136.160389] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1136.165808] LNet: Skipped 1 previous similar message [ 1136.472543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.496426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.528342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.562811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.604758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1136.673077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.723969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.776311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.822772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.854276] LNet: 59626:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.879080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.905120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.936722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.987780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.023975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.068712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.108901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1137.665889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_no_route -r 1 [ 1137.693813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_no_route -r 1 [ 1137.755917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1137.941155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.970511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.990381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.025449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.083184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1138.137598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.185332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.231240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.261324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.287534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.309296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.331908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.359551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.395202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.444109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.493818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.919241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_timeout -r 1 [ 1138.968506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_timeout -r 1 [ 1138.997282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1139.024545] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1139.034443] LNet: Skipped 2 previous similar messages [ 1139.244453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.271348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.288310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.302384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.319306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1139.351145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.370762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.401417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.421924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.434890] LNet: 59848:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.437908] LNet: 59848:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1139.465798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.516936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.553351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.584406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.600986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.645118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.690217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.100173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_error -r 1 [ 1140.123580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_error -r 1 [ 1140.158922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1140.328917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.363322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.413441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.444995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.478302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1140.518923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.547896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.572857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.599929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.660966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.713914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.750803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.783664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.829888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.871120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.903066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.098361] LNet: 60000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.104537] LNetError: 60000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.130291] LNet: Removed LNI 192.168.202.37@tcp [ 1142.538727] Key type .llcrypt unregistered [ 1142.540786] Key type ._llcrypt unregistered [ 1163.884772] Key type ._llcrypt registered [ 1163.889633] Key type .llcrypt registered [ 1165.394083] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:52:58 (1790898778) [ 1166.466709] Key type .llcrypt unregistered [ 1166.468867] Key type ._llcrypt unregistered [ 1173.670434] Key type ._llcrypt registered [ 1173.677129] Key type .llcrypt registered [ 1173.950863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1173.986705] alg: No test for adler32 (adler32-zlib) [ 1174.924877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1174.951188] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1182.858881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1194.405804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.475920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1194.524910] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1195.421823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1196.192253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_interrupt -r 1 [ 1196.246144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_interrupt -r 1 [ 1196.284622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_interrupt -r 1 [ 1196.325326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_interrupt -r 1 [ 1196.381823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_interrupt -r 1 [ 1196.453181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_interrupt -r 1 [ 1196.522928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_interrupt -r 1 [ 1196.583739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_interrupt -r 1 [ 1196.660260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1196.677403] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1196.687718] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp1 [ 1196.993867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.046972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.085790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.134636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.165333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1197.191512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.247685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.304712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.358553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.381809] LNet: 61475:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.403824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.452707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.487171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.514479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.548987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.608386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.672635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.151054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_dropped -r 1 [ 1198.193669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_dropped -r 1 [ 1198.222098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_dropped -r 1 [ 1198.260535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_dropped -r 1 [ 1198.302717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_dropped -r 1 [ 1198.338717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_dropped -r 1 [ 1198.398272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_dropped -r 1 [ 1198.433664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_dropped -r 1 [ 1198.472499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1198.491616] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1198.501973] LNet: Skipped 2 previous similar messages [ 1198.770262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.785738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1198.803518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.824413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.850902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1198.872826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.899570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.960570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.011264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.037898] LNet: 61598:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.078613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.143427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.187287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.233534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.276702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.303569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.344443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.008098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_aborted -r 1 [ 1200.054576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_aborted -r 1 [ 1200.094075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_aborted -r 1 [ 1200.134477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_aborted -r 1 [ 1200.210632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_aborted -r 1 [ 1200.279490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_aborted -r 1 [ 1200.348617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_aborted -r 1 [ 1200.377560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_aborted -r 1 [ 1200.412454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1200.434451] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1200.442492] LNet: Skipped 2 previous similar messages [ 1200.752413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.786923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.823598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.866828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.914193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1200.962497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.020456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.069899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.107380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.137577] LNet: 61722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.178719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.227247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.282488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.317566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.386920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.458949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.511732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.006151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_no_route -r 1 [ 1202.050598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_no_route -r 1 [ 1202.106245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_no_route -r 1 [ 1202.152651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_no_route -r 1 [ 1202.206843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_no_route -r 1 [ 1202.273445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_no_route -r 1 [ 1202.336700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_no_route -r 1 [ 1202.387297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_no_route -r 1 [ 1202.426840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1202.448345] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1202.454054] LNet: Skipped 2 previous similar messages [ 1202.700906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.738588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.774224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.802970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1202.832413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1202.870909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.922235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.956027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.993345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.027892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.065308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.103636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.146218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.194313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.227968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.258640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.776482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_timeout -r 1 [ 1203.808180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_timeout -r 1 [ 1203.860897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_timeout -r 1 [ 1203.902819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_timeout -r 1 [ 1203.975865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_timeout -r 1 [ 1204.041145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_timeout -r 1 [ 1204.103303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_timeout -r 1 [ 1204.159455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_timeout -r 1 [ 1204.233900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1204.440558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1204.474954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1204.514260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1204.567326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1204.616707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1204.651684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1204.683205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1204.729810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1204.784344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1204.815114] LNet: 61969:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1204.818557] LNet: 61969:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1204.857878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1204.919037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1204.969650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.010307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.041794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.081495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.143529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.872590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_error -r 1 [ 1205.926518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_error -r 1 [ 1205.992937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_error -r 1 [ 1206.061454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_error -r 1 [ 1206.113918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_error -r 1 [ 1206.179534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_error -r 1 [ 1206.222332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_error -r 1 [ 1206.268568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_error -r 1 [ 1206.358130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1206.700798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.730959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.765735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.801879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.831713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1206.879031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.924326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.963669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.008241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.057396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1207.118562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1207.172856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.224192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.256786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.291095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.341995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.245802] LNet: 62168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.256644] LNetError: 62168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.262481] LNet: Removed LNI 192.168.202.37@tcp [ 1213.279143] LNet: Removed LNI 192.168.202.37@tcp1 [ 1213.755241] Key type .llcrypt unregistered [ 1213.759115] Key type ._llcrypt unregistered [ 1229.536739] Key type ._llcrypt registered [ 1229.539277] Key type .llcrypt registered [ 1230.896966] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:54:03 (1790898843) [ 1232.383877] Key type .llcrypt unregistered [ 1232.386537] Key type ._llcrypt unregistered [ 1239.867564] Key type ._llcrypt registered [ 1239.871240] Key type .llcrypt registered [ 1240.182180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.210645] alg: No test for adler32 (adler32-zlib) [ 1241.073143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1241.096317] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1249.820258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1250.824885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_dropped -r 1 [ 1250.905326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_dropped -r 1 [ 1250.991623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1251.026156] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1251.359257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.417977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.460707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.507926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.537147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1251.568911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.601303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.647557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.680061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.697993] LNet: 63419:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.716276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.744801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.773749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.824542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.857269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.895747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.936803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1252.493651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_error -r 1 [ 1252.570216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_error -r 1 [ 1252.675031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1252.709979] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1253.068584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.093636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1253.121583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.183833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.228866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1253.270458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.320994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.383684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.424969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.445718] LNet: 63530:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.470929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.508257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.554264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.607448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.679903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.739652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.802899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1254.585424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_timeout -r 1 [ 1254.649697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_timeout -r 1 [ 1254.701139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1254.741409] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1255.166192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.202264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.236094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.270281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.308926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1255.338626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.366858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.410713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.450832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.469763] LNet: 63642:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.484587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.518865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.559444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.617855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.655157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.701781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.754250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.145520] LNet: 63683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.163675] LNetError: 63683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.203805] LNet: Removed LNI 192.168.202.37@tcp [ 1256.813310] Key type .llcrypt unregistered [ 1256.816064] Key type ._llcrypt unregistered [ 1274.460557] Key type ._llcrypt registered [ 1274.462379] Key type .llcrypt registered [ 1276.364188] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:54:48 (1790898888) [ 1277.697186] Key type .llcrypt unregistered [ 1277.701248] Key type ._llcrypt unregistered [ 1286.630092] Key type ._llcrypt registered [ 1286.632046] Key type .llcrypt registered [ 1286.906990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1286.935295] alg: No test for adler32 (adler32-zlib) [ 1287.917368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1287.946896] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1294.937518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1302.824588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.879493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.896643] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1304.010926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1304.625933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_dropped -r 1 [ 1304.659607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e remote_dropped -r 1 [ 1304.700951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_dropped -r 1 [ 1304.732863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e remote_dropped -r 1 [ 1304.765883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e remote_dropped -r 1 [ 1304.792986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e remote_dropped -r 1 [ 1304.821968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e remote_dropped -r 1 [ 1304.853735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e remote_dropped -r 1 [ 1304.888392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1304.910328] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1304.916829] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.137@tcp1 [ 1305.135847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.160884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.198579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.229845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.260171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1305.292963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.313437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.334645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.362826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.384822] LNet: 65156:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.397994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.424768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.452692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.486684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.542625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.557559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.572762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.593554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.976127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_error -r 1 [ 1306.007090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e remote_error -r 1 [ 1306.041812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_error -r 1 [ 1306.066739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e remote_error -r 1 [ 1306.101561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e remote_error -r 1 [ 1306.127602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e remote_error -r 1 [ 1306.203226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e remote_error -r 1 [ 1306.226727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e remote_error -r 1 [ 1306.267650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1306.284850] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1306.293515] LNet: Skipped 2 previous similar messages [ 1306.488423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.506525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1306.521341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.540491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.578441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1306.622457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.660080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.712702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.804914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.818804] LNet: 65282:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.845552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.871235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.902844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.963407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.028630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.128994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.186453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1307.204347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.673961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_timeout -r 1 [ 1307.700408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e remote_timeout -r 1 [ 1307.726297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_timeout -r 1 [ 1307.759413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e remote_timeout -r 1 [ 1307.790211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e remote_timeout -r 1 [ 1307.823517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e remote_timeout -r 1 [ 1307.889650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e remote_timeout -r 1 [ 1307.936194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e remote_timeout -r 1 [ 1307.994220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.137@tcp [ 1308.021099] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1308.169420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1308.183701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1308.205775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1308.223422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.241888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1308.262317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.286056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.316817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.350190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.368176] LNet: 65407:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.378695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.396665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.418189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.433340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.447493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.464732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.479452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1308.493324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1313.703680] LNet: 65485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.712758] LNetError: 65485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.719776] LNet: Removed LNI 192.168.202.37@tcp [ 1313.734317] LNet: Removed LNI 192.168.202.37@tcp1 [ 1314.168212] Key type .llcrypt unregistered [ 1314.171165] Key type ._llcrypt unregistered [ 1332.985853] Key type ._llcrypt registered [ 1332.988371] Key type .llcrypt registered [ 1334.699617] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:55:47 (1790898947) [ 1336.854773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1338.702546] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:55:51 (1790898951) [ 1339.566513] Key type .llcrypt unregistered [ 1339.570464] Key type ._llcrypt unregistered [ 1347.414902] Key type ._llcrypt registered [ 1347.423446] Key type .llcrypt registered [ 1347.731848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1347.750173] alg: No test for adler32 (adler32-zlib) [ 1348.731445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1348.757796] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1358.007959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1358.865941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e network_timeout -r 1 [ 1358.916028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e network_timeout -r 1 [ 1358.986665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1359.014150] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1359.237756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.268499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1359.301894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1359.357502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1359.407942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1359.458927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1359.514333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.578388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.659347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.694458] LNet: 67032:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.732592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.809521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.853473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.883674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.914694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.951370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.253260] LNet: 67071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.261047] LNetError: 67071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.279590] LNet: Removed LNI 192.168.202.37@tcp [ 1360.629540] Key type .llcrypt unregistered [ 1360.632065] Key type ._llcrypt unregistered [ 1382.006898] Key type ._llcrypt registered [ 1382.010558] Key type .llcrypt registered [ 1382.340146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1382.370422] alg: No test for adler32 (adler32-zlib) [ 1383.370457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1383.401975] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1392.646841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1402.484162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.553955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1402.595928] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1403.385904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1404.043802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e network_timeout -r 1 [ 1404.076580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e network_timeout -r 1 [ 1404.129764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e network_timeout -r 1 [ 1404.188077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e network_timeout -r 1 [ 1404.258841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e network_timeout -r 1 [ 1404.310897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e network_timeout -r 1 [ 1404.363814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e network_timeout -r 1 [ 1404.404296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e network_timeout -r 1 [ 1404.438900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1404.461051] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1404.642631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1404.661897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1404.690512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1404.721598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1404.756812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1404.787608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1404.817649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1404.846815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1404.875259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1404.894393] LNet: 67959:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1404.910859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1404.941963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1404.960667] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp1 [ 1404.966377] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.137@tcp1 [ 1404.979628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.005637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.027670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.043753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1409.363388] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1409.654874] LNet: 68033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.662573] LNetError: 68033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.669786] LNet: Removed LNI 192.168.202.37@tcp [ 1409.684523] LNet: Removed LNI 192.168.202.37@tcp1 [ 1409.999608] Key type .llcrypt unregistered [ 1410.002271] Key type ._llcrypt unregistered [ 1426.842637] Key type ._llcrypt registered [ 1426.850620] Key type .llcrypt registered [ 1428.285498] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:57:21 (1790899041) [ 1428.654864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1428.669540] alg: No test for adler32 (adler32-zlib) [ 1429.631666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.701626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1429.733103] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1430.496749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1430.518762] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1431.436277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 1431.503411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1431.558540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1431.610617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1431.651705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1431.688512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1431.710371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.37@tcp [ 1431.732598] LNetError: 69132:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.37@tcp to recovery [ 1432.735284] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp [ 1432.735620] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1434.783910] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1435.313373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1438.879465] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1439.933175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1439.965344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1440.036918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1440.054185] LNet: 69250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.059407] LNetError: 69250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.119179] LNet: Removed LNI 192.168.202.37@tcp [ 1441.121233] LNetError: 69250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.159835] LNet: Removed LNI 192.168.202.37@tcp1 [ 1441.198452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.273500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1441.332752] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1442.037960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1442.058320] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1442.760593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 1442.799571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1442.922846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1442.975467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1443.024928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1443.064973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1443.090280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1443.104450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.37@tcp [ 1443.117921] LNetError: 69522:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.37@tcp to recovery [ 1443.295774] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1445.669346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1449.440061] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1449.455321] LNet: Skipped 1 previous similar message [ 1454.721229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1454.777544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1454.814520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1462.315910] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:57:55 (1790899075) [ 1462.419852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1462.459547] LNet: 70238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.468802] LNetError: 70238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.475356] LNet: Removed LNI 192.168.202.37@tcp [ 1462.505104] LNet: Removed LNI 192.168.202.37@tcp1 [ 1462.553942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.611344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.642522] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1463.342379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.360825] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1464.396961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 1464.453346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1464.468963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1464.495861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1464.518635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1464.543365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1464.561304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.37@tcp --health 0 [ 1464.607993] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.37@tcp [ 1464.634871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.633757] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1465.650145] LNet: Skipped 1 previous similar message [ 1480.770212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1480.954031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1480.985966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.024573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.37@tcp --health 500 [ 1481.189712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1488.134276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1488.222643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.264674] LNet: 70732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.272666] LNetError: 70732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.276036] LNetError: 70732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1488.287927] LNet: Removed LNI 192.168.202.37@tcp [ 1488.298823] LNet: Removed LNI 192.168.202.37@tcp1 [ 1488.339429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.400531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1488.420995] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1489.467378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1489.499754] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1490.448528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 1490.472194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1490.542903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1490.571838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1490.615959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1490.667087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1490.695681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1490.747120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.37@tcp --health 0 [ 1491.425527] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1491.433843] LNet: Skipped 3 previous similar messages [ 1492.095884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1502.338449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1502.392433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1502.489410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1509.400804] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:58:42 (1790899122) [ 1509.952908] LNet: 71781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.961619] LNetError: 71781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.966886] LNetError: 71781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1509.975972] LNet: Removed LNI 192.168.202.37@tcp [ 1509.984593] LNet: Removed LNI 192.168.202.37@tcp1 [ 1510.662782] Key type .llcrypt unregistered [ 1510.665486] Key type ._llcrypt unregistered [ 1517.321648] Key type ._llcrypt registered [ 1517.325287] Key type .llcrypt registered [ 1517.512272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1517.532937] alg: No test for adler32 (adler32-zlib) [ 1518.508864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1518.561902] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1526.428562] Lustre: DEBUG MARKER: Initial discovery [ 1526.476971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.137@tcp [ 1528.319415] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1528.472342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1528.546270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.137@tcp [ 1528.584681] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1528.642272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1528.685823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.137@tcp -r 1 -m GET -l 3 [ 1528.748340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.137@tcp -r 1 -m GET -e local_error [ 1528.789575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.137@tcp -r 1 -m PUT -l 6 [ 1528.801990] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp [ 1530.102992] Lustre: DEBUG MARKER: Force oleg237-client.virtnet to discover 192.168.202.137@tcp (in background) [ 1530.273150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.137@tcp [ 1531.645845] Lustre: DEBUG MARKER: Force oleg237-server to discover 192.168.202.37@tcp [ 1533.348626] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1534.203368] Lustre: DEBUG MARKER: Wait for 72544 [ 1535.753132] Lustre: DEBUG MARKER: Finished wait on 72544 [ 1535.910522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1535.949366] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1535.965687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1536.357340] LNet: 72992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.362699] LNetError: 72992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.377094] LNet: Removed LNI 192.168.202.37@tcp [ 1536.802349] Key type .llcrypt unregistered [ 1536.810476] Key type ._llcrypt unregistered [ 1558.834453] Key type ._llcrypt registered [ 1558.839406] Key type .llcrypt registered [ 1560.273272] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:59:33 (1790899173) [ 1560.825282] Key type .llcrypt unregistered [ 1560.827688] Key type ._llcrypt unregistered [ 1567.661762] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1567.973914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1567.983123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1567.987470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1570.606892] Key type ._llcrypt registered [ 1570.609547] Key type .llcrypt registered [ 1570.751497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.762602] alg: No test for adler32 (adler32-zlib) [ 1571.773907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.849383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.894498] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1572.525754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1572.537672] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1583.951488] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:59:56 (1790899196) [ 1584.453932] LNet: 75070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.461257] LNetError: 75070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1584.464383] LNet: Removed LNI 10.1.2.3@tcp [ 1584.493668] LNet: Removed LNI 192.168.202.37@tcp [ 1585.497274] Key type .llcrypt unregistered [ 1585.501033] Key type ._llcrypt unregistered [ 1594.828626] Key type ._llcrypt registered [ 1594.838668] Key type .llcrypt registered [ 1595.050494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1595.068812] alg: No test for adler32 (adler32-zlib) [ 1596.215483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.369917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.414778] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1597.521986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1597.535981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1598.761946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.37@tcp [ 1599.844099] LNetError: Unexpected error -22 connecting to 192.168.202.37@tcp at host 192.168.202.37:988 [ 1607.175536] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:00:19 (1790899219) [ 1607.372757] LNet: 76193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.384861] LNetError: 76193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1607.407517] LNet: Removed LNI 10.1.2.3@tcp [ 1607.453058] LNet: Removed LNI 192.168.202.37@tcp [ 1608.404471] Key type .llcrypt unregistered [ 1608.406690] Key type ._llcrypt unregistered [ 1614.624560] Key type ._llcrypt registered [ 1614.629721] Key type .llcrypt registered [ 1614.852581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1614.878427] alg: No test for adler32 (adler32-zlib) [ 1615.910410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.991955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1616.028348] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1616.643862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1616.664200] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 1617.524886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.37@tcp1 --nid 192.168.202.37@tcp2 [ 1621.985424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp1 192.168.202.37@tcp2 [ 1622.000804] LNetError: 77035:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.37@tcp1 to 192.168.202.37@tcp2 - Route table may be misconfigured [ 1622.018870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp2 192.168.202.37@tcp1 [ 1628.540255] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:00:41 (1790899241) [ 1628.628546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.655746] LNet: 77600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.671888] LNetError: 77600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.727294] LNet: Removed LNI 192.168.202.37@tcp1 [ 1629.736126] LNetError: 77600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.772910] LNet: Removed LNI 192.168.202.37@tcp2 [ 1630.845836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.945891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.990409] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1632.204951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1632.245439] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1633.954935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 1634.006390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.37@tcp [ 1634.064405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp1 192.168.202.37@tcp1 [ 1634.109263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.37@tcp -d 192.168.202.37@tcp -e network_timeout [ 1634.175437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -e network_timeout [ 1634.284741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -e network_timeout [ 1634.342212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -e network_timeout [ 1634.447147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1634.511277] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1634.546486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1634.688695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1634.872267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1634.978368] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp [ 1642.621144] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:00:54 (1790899254) [ 1642.832824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.889970] LNet: 78467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1642.914421] LNetError: 78467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.935903] LNet: Removed LNI 192.168.202.37@tcp [ 1644.966990] LNet: Removed LNI 192.168.202.37@tcp1 [ 1645.117311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.294312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1645.759527] LNet: 78508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.414640] Key type .llcrypt unregistered [ 1646.416865] Key type ._llcrypt unregistered [ 1658.970879] Key type ._llcrypt registered [ 1658.975381] Key type .llcrypt registered [ 1660.482960] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:01:13 (1790899273) [ 1661.204516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1661.251644] alg: No test for adler32 (adler32-zlib) [ 1662.292935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.900865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1666.059377] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:01:18 (1790899278) [ 1666.164802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.228844] LNet: 79612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.263469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.323795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.347871] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1667.098791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1667.145560] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1668.218268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1668.286285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp1 [ 1668.350956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp1 [ 1673.938343] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:01:27 (1790899287) [ 1675.672824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg237-server [ 1677.423517] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:01:30 (1790899290) [ 1678.973908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg237-server [ 1680.633433] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:01:33 (1790899293) [ 1682.336770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg237-server [ 1684.393396] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:01:37 (1790899297) [ 1686.573750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg237-server [ 1688.327542] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:01:41 (1790899301) [ 1689.816086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg237-server [ 1691.708779] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:01:44 (1790899304) [ 1693.845443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg237-server [ 1695.835969] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:01:48 (1790899308) [ 1697.465094] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg237-server [ 1699.337516] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:01:52 (1790899312) [ 1701.358732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg237-server [ 1702.914561] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:01:56 (1790899316) [ 1705.132511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg237-server [ 1706.571907] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:01:59 (1790899319) [ 1706.819297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.850288] LNet: 83496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.859445] LNetError: 83496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.866762] LNet: Removed LNI 192.168.202.37@tcp [ 1707.877967] LNet: Removed LNI 192.168.202.37@tcp1 [ 1707.926717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.987518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.032956] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1709.031380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1709.123780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1710.581424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.619391] LNet: 83652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.626371] LNetError: 83652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.634264] LNetError: 83652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1710.665332] LNet: Removed LNI 192.168.202.37@tcp [ 1710.706453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.798740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1710.830764] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1711.974757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1712.183736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.216422] LNetError: 83798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.241071] LNet: Removed LNI 192.168.202.37@tcp [ 1712.308414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.367407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.409035] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1713.865148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1713.995916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1714.276942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.314669] LNet: 83945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.325893] LNet: 83945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1714.340424] LNetError: 83945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.373895] LNet: Removed LNI 192.168.202.37@tcp [ 1714.450588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.524566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1714.544854] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1715.489674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1715.620663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.727669] LNet: Removed LNI 192.168.202.37@tcp [ 1716.846474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.988312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.020522] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1718.281097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1718.452594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1718.834370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.867540] LNet: 84237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.875368] LNet: 84237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1718.885269] LNetError: 84237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.893309] LNetError: 84237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.921603] LNet: Removed LNI 192.168.202.37@tcp [ 1718.968041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.066830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1719.094313] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1720.292729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1720.676821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.774792] LNet: Removed LNI 192.168.202.37@tcp [ 1721.870360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.994066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.063636] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1723.366661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1723.508781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1723.826531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.903472] LNet: Removed LNI 192.168.202.37@tcp [ 1725.003395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.091346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1725.127419] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1726.301192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1726.618871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.668040] LNet: Removed LNI 192.168.202.37@tcp [ 1727.746936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.832415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.864800] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1728.665455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1728.823841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1729.038912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.059386] LNet: 84823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.061411] LNet: 84823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1729.073893] LNetError: 84823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.079701] LNetError: 84823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1730.169029] LNet: Removed LNI 192.168.202.37@tcp [ 1730.227975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.346625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1730.388777] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1731.011531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1731.318866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.397339] LNet: Removed LNI 192.168.202.37@tcp [ 1732.413341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.448974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.493727] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1733.136190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1733.177928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1733.345234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.379473] LNet: Removed LNI 192.168.202.37@tcp [ 1734.493883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.575343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1734.606566] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1735.353246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 1735.561786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.614596] LNet: Removed LNI 192.168.202.37@tcp [ 1736.685173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.797476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.850545] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1737.568429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1737.620783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.654923] LNet: Removed LNI 192.168.202.37@tcp [ 1737.683381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.770188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.797787] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1738.497198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1738.553720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.576344] LNet: Removed LNI 192.168.202.37@tcp [ 1738.598890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.697803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1738.782868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.809081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.861199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1738.934696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.978251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.055230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1739.149884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.236232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.355378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1739.386560] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1740.395548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.464156] LNet: Removed LNI 192.168.202.37@tcp [ 1740.498611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.630713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1749.432359] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:02:42 (1790899362) [ 1749.540366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.595548] LNet: 86275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.599617] LNet: 86275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1749.619576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.692666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1749.740030] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/180] [ 1751.422919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.467331] LNetError: 86420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.473262] LNetError: 86420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1751.491775] LNet: Removed LNI 192.168.202.37@tcp231 [ 1751.541749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.679106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1751.710456] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/180] [ 1751.717696] LNet: 86428:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1752.070272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1752.087849] LNet: Removed LNI 192.168.202.37@tcp231 [ 1752.097800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1752.112091] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/180] [ 1753.177698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.199586] LNet: Removed LNI 192.168.202.37@tcp231 [ 1753.217045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.289429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1753.307974] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/0] [ 1753.312467] LNet: 86580:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1753.887334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1753.910130] LNet: Removed LNI 192.168.202.37@tcp231 [ 1753.937304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1753.990261] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/0] [ 1755.505098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.552934] LNet: Removed LNI 192.168.202.37@tcp231 [ 1755.600332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.663290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1755.699258] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/60] [ 1755.715216] LNet: 86732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1756.054823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1756.088582] LNet: Removed LNI 192.168.202.37@tcp231 [ 1756.100736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1756.147862] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/60] [ 1757.278080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.321482] LNet: Removed LNI 192.168.202.37@tcp231 [ 1757.360694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.453204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1757.506274] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/180] [ 1758.197348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1758.253777] LNet: Removed LNI 192.168.202.37@tcp231 [ 1758.289531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1758.351276] LNet: Added LNI 192.168.202.37@tcp231 [8/256/0/180] [ 1766.972500] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:03:00 (1790899380) [ 1767.070752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.141541] LNet: Removed LNI 192.168.202.37@tcp231 [ 1767.196659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.331472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1767.358631] LNet: Added LNI 192.168.202.37@tcp232 [8/256/0/180] [ 1768.069295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1768.100842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1768.127377] LNet: Removed LNI 192.168.202.37@tcp232 [ 1768.148976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1768.191518] LNet: Added LNI 192.168.202.37@tcp232 [8/256/0/180] [ 1768.947153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1768.992946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1769.026758] LNet: Removed LNI 192.168.202.37@tcp232 [ 1769.065385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1769.115728] LNet: Added LNI 192.168.202.37@tcp232 [8/256/0/180] [ 1769.754772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1769.790550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1769.825567] LNet: Removed LNI 192.168.202.37@tcp232 [ 1769.855851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1769.904745] LNet: Added LNI 192.168.202.37@tcp232 [8/256/0/180] [ 1770.707337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1770.751735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1770.781771] LNet: Removed LNI 192.168.202.37@tcp232 [ 1770.791286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1777.102855] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:03:10 (1790899390) [ 1778.068986] Key type .llcrypt unregistered [ 1778.071302] Key type ._llcrypt unregistered [ 1785.894689] Key type ._llcrypt registered [ 1785.897055] Key type .llcrypt registered [ 1786.300366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1786.329748] alg: No test for adler32 (adler32-zlib) [ 1787.373886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1787.437334] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1797.191290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1808.358937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.388794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1808.416526] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1809.227251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1810.012313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_interrupt -r 2 [ 1810.054488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_interrupt -r 2 [ 1810.118948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_interrupt -r 2 [ 1810.164500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_interrupt -r 2 [ 1810.209897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_interrupt -r 2 [ 1810.256967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_interrupt -r 2 [ 1810.286406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_interrupt -r 2 [ 1810.360184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_interrupt -r 2 [ 1810.440224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.137@tcp [ 1810.489255] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1810.505761] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp [ 1810.885549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1810.921835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1810.963558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1811.037192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1811.120075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1811.199494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1811.257248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1811.294647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1811.338645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1811.399893] LNet: 89391:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1811.442689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1811.520584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1811.579117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1811.619521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1811.652849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.710206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1812.230506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_dropped -r 2 [ 1812.265990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_dropped -r 2 [ 1812.313410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_dropped -r 2 [ 1812.349560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_dropped -r 2 [ 1812.428572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_dropped -r 2 [ 1812.485903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_dropped -r 2 [ 1812.542607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_dropped -r 2 [ 1812.603732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_dropped -r 2 [ 1812.651646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.137@tcp [ 1812.677824] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1812.993825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.039072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1813.096986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.166040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.200883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1813.271800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.311473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.349436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.393784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.433418] LNet: 89513:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.460725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1813.496721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1813.559785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1813.615756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1813.663503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1813.703329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1814.303699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_aborted -r 2 [ 1814.365924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_aborted -r 2 [ 1814.448764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_aborted -r 2 [ 1814.507235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_aborted -r 2 [ 1814.568949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_aborted -r 2 [ 1814.632322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_aborted -r 2 [ 1814.663201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_aborted -r 2 [ 1814.702981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_aborted -r 2 [ 1814.751374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.137@tcp [ 1814.792205] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1815.127260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1815.162527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1815.208892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1815.249202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1815.288757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1815.355108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1815.435738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1815.495874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1815.550703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1815.567893] LNet: 89636:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1815.605487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1815.660468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1815.732135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1815.794291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1815.870976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1815.936200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1816.610708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_no_route -r 2 [ 1816.652190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_no_route -r 2 [ 1816.692648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_no_route -r 2 [ 1816.744616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_no_route -r 2 [ 1816.781987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_no_route -r 2 [ 1816.835303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_no_route -r 2 [ 1816.872826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_no_route -r 2 [ 1816.906653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_no_route -r 2 [ 1816.943267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.137@tcp [ 1816.960727] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1817.238989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1817.281524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1817.325665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1817.360172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1817.415113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1817.457518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1817.526690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1817.563567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1817.616736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1817.643085] LNet: 89758:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1817.668415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1817.707038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1817.748601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1817.781503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1817.814383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1817.847564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1818.524721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e local_timeout -r 2 [ 1818.573370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e local_timeout -r 2 [ 1818.599851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e local_timeout -r 2 [ 1818.626716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e local_timeout -r 2 [ 1818.657535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e local_timeout -r 2 [ 1818.683348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e local_timeout -r 2 [ 1818.708658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e local_timeout -r 2 [ 1818.753685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e local_timeout -r 2 [ 1818.796476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.137@tcp [ 1819.154859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.213842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.276534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.335906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.396876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1819.459945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.496982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.543289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.595290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.644575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.679508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.732695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1819.758170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.803640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.863889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.570572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp -m GET -e remote_dropped -r 2 [ 1820.623766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp -d 192.168.202.137@tcp1 -m GET -e remote_dropped -r 2 [ 1820.669663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp -m GET -e remote_dropped -r 2 [ 1820.748901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp -d 192.168.202.37@tcp1 -m GET -e remote_dropped -r 2 [ 1820.833461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp -m GET -e remote_dropped -r 2 [ 1820.903738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.137@tcp1 -m GET -e remote_dropped -r 2 [ 1820.960483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp -m GET -e remote_dropped -r 2 [ 1821.014419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.37@tcp1 -d 192.168.202.37@tcp1 -m GET -e remote_dropped -r 2 [ 1821.052526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.37@tcp 192.168.202.137@tcp [ 1821.077594] LNet: There was an unexpected network error while writing to 192.168.202.137: rc = -22 [ 1821.085841] LNet: Skipped 1 previous similar message [ 1821.090886] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.137@tcp [ 1821.444470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.480889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1821.526455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.591667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.652225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1821.710533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.782206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.846465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.885731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.919713] LNet: 90002:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1821.930383] LNet: 90002:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1821.961327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1822.049127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1822.122158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1822.158946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1822.207889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1822.250471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.395690] LNet: 90077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.401511] LNetError: 90077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.408159] LNet: Removed LNI 192.168.202.37@tcp [ 1828.423320] LNet: Removed LNI 192.168.202.37@tcp1 [ 1828.785865] Key type .llcrypt unregistered [ 1828.789739] Key type ._llcrypt unregistered [ 1848.015814] Key type ._llcrypt registered [ 1848.017696] Key type .llcrypt registered [ 1849.569961] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:04:22 (1790899462) [ 1850.111769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.139287] alg: No test for adler32 (adler32-zlib) [ 1851.166862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.253407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.290229] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1852.237954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1852.263822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.37@tcp [ 1852.305189] LNetError: 91055:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.37@tcp to recovery [ 1852.405597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.37@tcp [ 1852.537456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.516697] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:04:32 (1790899472) [ 1859.828289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.869787] LNet: 91645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.880175] LNetError: 91645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.908703] LNet: Removed LNI 192.168.202.37@tcp [ 1859.938515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.000937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.020517] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1860.840402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1860.872288] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 1861.895732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.37@tcp [ 1861.912493] LNetError: 91912:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.202.37@tcp to recovery [ 1862.048371] LNet: 1 local NIs in recovery (showing 1): 192.168.202.37@tcp [ 1862.052603] LNet: There was an unexpected network error while writing to 192.168.202.37: rc = -22 [ 1863.241780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.37@tcp [ 1871.102422] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:04:44 (1790899484) [ 1872.897986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg237-server [ 1874.896352] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:04:47 (1790899487) [ 1877.030555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg237-server [ 1879.066278] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:04:51 (1790899491) [ 1881.199765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg237-server [ 1883.117800] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:04:56 (1790899496) [ 1883.278755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1883.321572] LNet: 93532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.329504] LNetError: 93532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.337404] LNet: Removed LNI 192.168.202.37@tcp [ 1883.351237] LNet: Removed LNI 192.168.202.37@tcp1 [ 1883.373140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1883.436396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1883.479051] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1884.158308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1884.219612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1884.255783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1884.346961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1884.395037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1884.456393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1889.951144] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:05:03 (1790899503) [ 1890.052795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.089844] LNet: 94258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.103522] LNetError: 94258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.112479] LNetError: 94258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1890.142232] LNet: Removed LNI 192.168.202.37@tcp [ 1890.169660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.234485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1890.279980] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1896.773379] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:05:09 (1790899509) [ 1896.959976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.992112] LNet: 94968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.999816] LNetError: 94968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.009973] LNet: Removed LNI 192.168.202.37@tcp [ 1897.024436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.056686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1897.078284] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1904.641477] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:05:17 (1790899517) [ 1906.576243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1907.866211] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:05:21 (1790899521) [ 1908.610592] LNet: 96002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.619523] LNetError: 96002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.640432] LNet: Removed LNI 192.168.202.37@tcp [ 1909.291603] Key type .llcrypt unregistered [ 1909.298672] Key type ._llcrypt unregistered [ 1916.630970] Key type ._llcrypt registered [ 1916.633200] Key type .llcrypt registered [ 1916.792979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1916.813567] alg: No test for adler32 (adler32-zlib) [ 1917.736587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1917.763040] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1925.744253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 1931.536121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.137@tcp [ 1931.935291] LNet: 96419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.938770] LNetError: 96419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.949578] LNet: Removed LNI 192.168.202.37@tcp [ 1932.280650] Key type .llcrypt unregistered [ 1932.282737] Key type ._llcrypt unregistered [ 1942.838185] Key type ._llcrypt registered [ 1942.839608] Key type .llcrypt registered [ 1944.023325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1945.536403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1946.728466] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:06:00 (1790899560) [ 1948.323431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1949.584685] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:06:03 (1790899563) [ 1951.312578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg237-server [ 1953.134436] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:06:06 (1790899566) [ 1954.852908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg237-server [ 1956.335201] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:06:09 (1790899569) [ 1958.625422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.842094] alg: No test for adler32 (adler32-zlib) [ 1960.043157] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 1960.389430] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 1962.207419] Key type lgssc registered [ 1963.810741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1988.596788] Key type lgssc unregistered [ 1988.970625] LNet: 99602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.003806] LNetError: 99602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.047074] LNet: Removed LNI 192.168.202.37@tcp [ 1989.866811] Key type .llcrypt unregistered [ 1989.869270] Key type ._llcrypt unregistered [ 2007.036785] Key type ._llcrypt registered [ 2007.038181] Key type .llcrypt registered [ 2008.024715] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:07:01 (1790899621) [ 2008.783702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.792396] alg: No test for adler32 (adler32-zlib) [ 2009.774930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.855211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2009.901287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2009.936973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2009.973413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2017.557198] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:07:10 (1790899630) [ 2017.689156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.730051] LNet: 101020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.757184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.871862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2017.909255] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2018.942914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.961372] LNet: 101170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.969878] LNetError: 101170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.980187] LNet: Removed LNI 192.168.202.37@tcp [ 2018.990695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.087105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2019.134558] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2019.868526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.888515] LNetError: 101198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.910723] LNet: Removed LNI 192.168.202.37@tcp [ 2019.963772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.078363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2020.131445] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2027.301534] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:07:20 (1790899640) [ 2027.375741] LNet: 101783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.383463] LNet: 101783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2027.390978] LNetError: 101783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.420247] LNet: Removed LNI 192.168.202.37@tcp [ 2028.049532] Key type .llcrypt unregistered [ 2028.053495] Key type ._llcrypt unregistered [ 2035.407540] Key type ._llcrypt registered [ 2035.411187] Key type .llcrypt registered [ 2035.654059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2035.684587] alg: No test for adler32 (adler32-zlib) [ 2036.751076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.882473] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2036.889991] LNet: *** cfs_fail_loc=e003, val=100*** [ 2037.522454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2037.537204] LNet: *** cfs_fail_loc=e003, val=100*** [ 2037.539860] LNet: Skipped 2 previous similar messages [ 2037.545405] Ping reply received with mapping: [ 2037.547590] found metadata reply NI[192.168.202.37@tcp] [ 2037.549912] 25 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2037.552703] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2037.555099] 5a 5a 5a 5a 5a 5a 5a 5a [ 2037.660516] LNet: 102185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.665208] LNetError: 102185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.703597] LNet: Removed LNI 192.168.202.37@tcp [ 2039.398708] Key type .llcrypt unregistered [ 2039.401727] Key type ._llcrypt unregistered [ 2051.778552] Key type ._llcrypt registered [ 2051.780246] Key type .llcrypt registered [ 2052.950742] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:07:46 (1790899666) [ 2053.572394] Key type .llcrypt unregistered [ 2053.574621] Key type ._llcrypt unregistered [ 2059.905769] Key type ._llcrypt registered [ 2059.909949] Key type .llcrypt registered [ 2060.312970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2060.349806] alg: No test for adler32 (adler32-zlib) [ 2061.730248] LNetError: 103227:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2062.243623] Key type .llcrypt unregistered [ 2062.245751] Key type ._llcrypt unregistered [ 2067.233104] Key type ._llcrypt registered [ 2067.234229] Key type .llcrypt registered [ 2068.288576] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:08:01 (1790899681) [ 2068.845677] Key type .llcrypt unregistered [ 2068.848232] Key type ._llcrypt unregistered [ 2075.402683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2075.404792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2077.932760] Key type ._llcrypt registered [ 2077.935331] Key type .llcrypt registered [ 2078.161713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2078.198098] alg: No test for adler32 (adler32-zlib) [ 2079.216891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.275679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2088.568451] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:08:21 (1790899701) [ 2088.634337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.675434] LNet: 104999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.683264] LNetError: 104999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2088.721032] LNet: Removed LNI 10.1.2.3@tcp [ 2088.830105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.486720] Key type .llcrypt unregistered [ 2089.490697] Key type ._llcrypt unregistered [ 2095.597545] Key type ._llcrypt registered [ 2095.604280] Key type .llcrypt registered [ 2095.803458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2095.820771] alg: No test for adler32 (adler32-zlib) [ 2096.860242] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2096.951862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2097.171726] LNet: 105294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.175938] LNetError: 105294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.226981] LNet: Removed LNI 192.168.202.37@tcp [ 2099.117420] Key type .llcrypt unregistered [ 2099.120452] Key type ._llcrypt unregistered [ 2111.444493] Key type ._llcrypt registered [ 2111.447501] Key type .llcrypt registered [ 2112.842987] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:08:45 (1790899725) [ 2113.503414] Key type .llcrypt unregistered [ 2113.511297] Key type ._llcrypt unregistered [ 2119.462106] Key type ._llcrypt registered [ 2119.465913] Key type .llcrypt registered [ 2119.739636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.756477] alg: No test for adler32 (adler32-zlib) [ 2126.681318] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:08:59 (1790899739) [ 2126.799612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.861371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.955536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.992318] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2127.808608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.875276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2127.941574] LNet: Removed LNI 192.168.202.37@tcp [ 2128.233763] LNet: 107038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.753546] Key type .llcrypt unregistered [ 2128.760248] Key type ._llcrypt unregistered [ 2140.305396] Key type ._llcrypt registered [ 2140.307776] Key type .llcrypt registered [ 2141.857403] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:09:14 (1790899754) [ 2142.449028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2142.462891] alg: No test for adler32 (adler32-zlib) [ 2143.468945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.508546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.524282] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2144.197704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2144.267358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.37@tcp [ 2144.296821] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2144.299941] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2144.302670] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2144, 0, 0 [ 2144.307305] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2144.309989] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2144.313347] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2144.316279] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2144, 0, 0 [ 2144.321345] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2144.323095] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2144.327406] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2144.330853] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2144, 0, 2194 [ 2144.335206] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2144.337891] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2144.342729] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2144.345930] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2144, 0, 2194 [ 2144.349720] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2151.518995] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:09:24 (1790899764) [ 2151.950089] LNet: 108576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.960586] LNetError: 108576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.998261] LNet: Removed LNI 192.168.202.37@tcp [ 2153.805478] Key type .llcrypt unregistered [ 2153.810863] Key type ._llcrypt unregistered [ 2160.468503] Key type ._llcrypt registered [ 2160.474080] Key type .llcrypt registered [ 2160.817604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2160.828414] alg: No test for adler32 (adler32-zlib) [ 2161.803461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2161.846869] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2168.303795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 2178.033803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.103811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2178.148976] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 2179.001875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 2179.167909] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2179.514064] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2179.519510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2179.523936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2181.894938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2181.951355] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2193.623885] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:10:06 (1790899806) [ 2193.855193] LNet: 110089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.861368] LNetError: 110089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.866528] LNet: Removed LNI 192.168.202.37@tcp [ 2194.911193] LNet: Removed LNI 192.168.202.37@tcp1 [ 2194.916415] LNetError: 110089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2194.929760] LNetError: 110089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2194.941828] LNet: Removed LNI 10.1.2.3@tcp99 [ 2195.449996] Key type .llcrypt unregistered [ 2195.451599] Key type ._llcrypt unregistered [ 2202.455183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2202.457378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2205.136105] Key type ._llcrypt registered [ 2205.139841] Key type .llcrypt registered [ 2205.318799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2205.333471] alg: No test for adler32 (adler32-zlib) [ 2206.185116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.263843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.300982] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2207.071072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2207.095798] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2208.596773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2208.621186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2208.647897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2208.664023] LNetError: 110832:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2208.707393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2208.759310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2209.086056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2209.129767] LNet: 110852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.132818] LNetError: 110852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2209.135498] LNet: Removed LNI 10.1.2.3@tcp [ 2210.162546] LNet: Removed LNI 192.168.202.37@tcp [ 2210.795317] Key type .llcrypt unregistered [ 2210.796599] Key type ._llcrypt unregistered [ 2221.013599] Key type ._llcrypt registered [ 2221.016988] Key type .llcrypt registered [ 2222.251836] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:10:35 (1790899835) [ 2223.073461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.086915] alg: No test for adler32 (adler32-zlib) [ 2224.177855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.234230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.255945] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2224.990691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2225.101268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg237-client@tcp [ 2229.850577] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:10:43 (1790899843) [ 2230.328899] LNet: 112384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2230.345097] LNetError: 112384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.377307] LNet: Removed LNI 192.168.202.37@tcp [ 2231.105845] Key type .llcrypt unregistered [ 2231.109621] Key type ._llcrypt unregistered [ 2237.297228] Key type ._llcrypt registered [ 2237.323750] Key type .llcrypt registered [ 2237.670497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2237.728513] alg: No test for adler32 (adler32-zlib) [ 2238.634947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2238.658871] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2238.714705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2251.167687] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:11:04 (1790899864) [ 2254.904737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2259.031344] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:11:11 (1790899871) [ 2261.010336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2263.147674] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:11:15 (1790899875) [ 2263.385092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.428400] LNet: 113808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.439930] LNetError: 113808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.450971] LNet: Removed LNI 192.168.202.37@tcp [ 2263.469703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.557120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.604169] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2264.360664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2264.462264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2272.703438] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:11:25 (1790899885) [ 2273.178918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.223471] LNet: 114517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.238769] LNetError: 114517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.268568] LNet: Removed LNI 192.168.202.37@tcp [ 2273.342604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.408420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2273.431642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2274.104173] Key type .llcrypt unregistered [ 2274.105457] Key type ._llcrypt unregistered [ 2286.880446] Key type ._llcrypt registered [ 2286.884233] Key type .llcrypt registered [ 2288.533184] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:11:41 (1790899901) [ 2289.086656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.101708] alg: No test for adler32 (adler32-zlib) [ 2290.095971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.186123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.267871] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2291.071784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2291.181333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2298.597476] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:11:51 (1790899911) [ 2298.759263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.824850] LNet: 116064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.851531] LNetError: 116064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.929655] LNet: Removed LNI 192.168.202.37@tcp [ 2299.024965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.136677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2299.216519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2307.480680] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:12:00 (1790899920) [ 2307.601435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.654083] LNet: 116647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.713708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.853934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2308.371111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2308.463294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2308.588088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2309.132635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2309.157247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2309.233103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.248227] LNet: 116686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.265345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.331263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2309.747391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2309.801557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2309.895596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2310.398082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2310.454945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2317.242622] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:12:10 (1790899930) [ 2317.391291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.424806] LNet: 117284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.498927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.553599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.587467] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2318.336777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2318.358295] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2319.253410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2319.345396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2319.538936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.557516] LNetError: 117566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2319.563417] LNet: Removed LNI 192.168.202.37@tcp [ 2319.574758] LNet: Removed LNI 192.168.202.37@tcp2 [ 2319.590624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.637226] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2319.644347] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2319.656359] LNet: 117572:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2320.314552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2320.494400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.567164] LNet: Removed LNI 192.168.202.37@tcp [ 2321.570734] LNetError: 117609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.581996] LNetError: 117609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2321.608607] LNet: Removed LNI 192.168.202.37@tcp2 [ 2321.640494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.687656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2321.717496] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2322.490305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2322.516671] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2323.488628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2323.562446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2323.635160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2323.908021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.932321] LNet: 117893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.934979] LNet: 117893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2323.940134] LNetError: 117893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.959153] LNet: Removed LNI 192.168.202.37@tcp [ 2325.998229] LNet: Removed LNI 192.168.202.37@tcp2 [ 2326.022800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.149234] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2326.158822] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2326.167573] LNet: 117899:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2326.741936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2327.019342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.051587] LNet: Removed LNI 192.168.202.37@tcp [ 2328.101929] LNet: Removed LNI 192.168.202.37@tcp2 [ 2328.182053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.251375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.286103] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2329.148145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2329.180079] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2329.918493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2329.974348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2330.192963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.212573] LNetError: 118216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.218567] LNetError: 118216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2330.224632] LNet: Removed LNI 192.168.202.37@tcp [ 2330.251901] LNet: Removed LNI 192.168.202.37@tcp2 [ 2330.337966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.442755] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2330.450797] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2330.461533] LNet: 118223:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2331.130973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2331.390130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.447172] LNet: Removed LNI 192.168.202.37@tcp [ 2332.467401] LNet: Removed LNI 192.168.202.37@tcp2 [ 2332.528513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.617818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2332.687353] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2333.704839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2333.722372] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2334.810161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2334.861608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2334.934784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2335.214830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.239405] LNet: 118545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.244826] LNet: 118545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2336.289137] LNet: Removed LNI 192.168.202.37@tcp [ 2337.326956] LNet: Removed LNI 192.168.202.37@tcp2 [ 2337.376949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.516514] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2337.523841] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2337.534441] LNet: 118551:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.326763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2338.582988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2338.621527] LNetError: 118588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.625962] LNetError: 118588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2338.630734] LNet: Removed LNI 192.168.202.37@tcp [ 2339.687413] LNet: Removed LNI 192.168.202.37@tcp2 [ 2339.737236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.841147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2339.878819] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2340.640484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2340.658309] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2341.477590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2341.524842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2341.625973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2342.159831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.199279] LNet: Removed LNI 192.168.202.37@tcp [ 2343.221266] LNet: Removed LNI 192.168.202.37@tcp2 [ 2343.268459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.413322] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2343.423895] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2343.432790] LNet: 118891:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.026349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2344.442572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.504731] LNet: Removed LNI 192.168.202.37@tcp [ 2345.515398] LNet: Removed LNI 192.168.202.37@tcp2 [ 2345.562924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.647717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.684814] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2346.393371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2346.411943] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2347.248818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2347.290735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2347.325582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2347.385885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2347.926784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.961606] LNet: Removed LNI 192.168.202.37@tcp [ 2349.989303] LNet: Removed LNI 192.168.202.37@tcp2 [ 2350.064818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.174527] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2350.187056] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2350.859216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2357.613452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2359.039185] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:12:52 (1790899972) [ 2359.215687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.229816] LNet: 120009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.232242] LNet: 120009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2359.235858] LNetError: 120009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.238859] LNetError: 120009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2359.242986] LNet: Removed LNI 192.168.202.37@tcp [ 2360.295487] LNet: Removed LNI 192.168.202.37@tcp2 [ 2360.335268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.615824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.631268] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2361.222626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2361.243486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2362.129799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.173889] LNet: Removed LNI 10.1.2.3@tcp [ 2362.182409] LNet: Removed LNI 192.168.202.37@tcp [ 2362.225872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.316661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2362.346545] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2362.354548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2362.662805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2362.691434] LNet: Removed LNI 192.168.202.37@tcp [ 2362.705922] LNet: Removed LNI 10.1.2.3@tcp [ 2368.215478] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 20:13:01 (1790899981) [ 2368.610382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.716443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.798224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2368.827056] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2369.459855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2369.489056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2369.549899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.580710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.613898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.645883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.665988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.687756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.712371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.746767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.781417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.813965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2369.968773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2375.504010] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 20:13:08 (1790899988) [ 2375.602897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2376.681072] LNet: Removed LNI 192.168.202.37@tcp [ 2376.747700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.810723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2376.836933] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2377.628971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2377.659885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2377.691613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.731500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.764529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.798699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.835988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.889917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.928652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2377.966655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.004760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.062212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.145973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2378.196845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.254572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.298955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.346988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.394751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.442934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.482139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.515756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.552500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2378.597239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.37@tcp [ 2384.914513] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:13:17 (1790899997) [ 2385.083320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.154833] LNet: Removed LNI 192.168.202.37@tcp [ 2386.197983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.391856] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2386.528084] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2386.541688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2386.550758] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2389.032879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2389.077736] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2389.785365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2389.803914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2391.920597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2391.992442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2392.084990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.37@tcp -m PUT -e local_timeout -r 1 [ 2392.146154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2392.219512] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2392.232537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2392.470108] LNetError: 122448:0:(lib-move.c:4794:lnet_parse()) 192.168.202.37@tcp, src 192.168.202.37@tcp: Dropping PUT (error -108 looking up sender) [ 2393.503702] LNet: 122905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.507751] LNet: 122905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2393.514362] LNetError: 122905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2393.517371] LNetError: 122905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2393.533184] LNet: Removed LNI 10.1.2.3@tcp [ 2394.602623] LNet: Removed LNI 192.168.202.37@tcp [ 2395.008317] Key type .llcrypt unregistered [ 2395.010843] Key type ._llcrypt unregistered [ 2406.776261] Key type ._llcrypt registered [ 2406.778516] Key type .llcrypt registered [ 2408.112493] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:13:41 (1790900021) [ 2408.391946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2408.402573] alg: No test for adler32 (adler32-zlib) [ 2409.283472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.695369] LNet: 123758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.179351] Key type .llcrypt unregistered [ 2410.183338] Key type ._llcrypt unregistered [ 2417.844280] Key type ._llcrypt registered [ 2417.845536] Key type .llcrypt registered [ 2418.168828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2418.212680] alg: No test for adler32 (adler32-zlib) [ 2419.282404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2419.327901] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2426.988938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 2427.208285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2427.836202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.137@tcp [ 2428.108323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2428.142813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2428.177295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2428.218985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2428.258422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2428.299739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2428.346384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2428.415393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2428.477280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2428.500476] LNet: 124209:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2428.547547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2428.610570] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.137@tcp [ 2428.648065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2428.696294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2428.756498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2428.798373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2428.858780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2430.155869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2430.638629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2430.718069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2430.828192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2430.946850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2431.004958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2431.070446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2431.136485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.226290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.323836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.383546] LNet: 124328:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.447966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2431.561202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2431.615426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2431.686322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2431.794547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.871395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2433.264848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.137@tcp [ 2433.778432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.878678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2433.944443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2434.023356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2434.104161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2434.167024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2434.250598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2434.326632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2434.387355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2434.418230] LNet: 124447:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.472838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.526920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.597772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2434.705815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2434.781429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2434.834290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.996460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2436.492833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.558162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2436.625843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2436.680243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2436.727610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2436.787523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2436.862759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2436.981127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2437.046319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2437.096535] LNet: 124567:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2437.125438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2437.187336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2437.272407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2437.367564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2437.428555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2437.492929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2437.998571] LNet: 124613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.007267] LNetError: 124613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.034127] LNet: Removed LNI 192.168.202.37@tcp [ 2438.676246] Key type .llcrypt unregistered [ 2438.677861] Key type ._llcrypt unregistered [ 2458.112618] Key type ._llcrypt registered [ 2458.114832] Key type .llcrypt registered [ 2459.192816] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:14:32 (1790900072) [ 2459.661029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2459.671506] alg: No test for adler32 (adler32-zlib) [ 2460.659227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.127753] LNet: 125500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.765489] Key type .llcrypt unregistered [ 2461.767473] Key type ._llcrypt unregistered [ 2469.050756] Key type ._llcrypt registered [ 2469.053752] Key type .llcrypt registered [ 2469.546816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2469.581558] alg: No test for adler32 (adler32-zlib) [ 2470.725350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2470.771094] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2479.551904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 2490.837521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2490.872117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2490.906361] LNet: Added LNI 192.168.202.37@tcp1 [8/256/0/180] [ 2492.007342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.137@tcp [ 2492.221585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2493.067612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.137@tcp [ 2493.332475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2493.343241] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.137@tcp [ 2493.411940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2493.504603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2493.584665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2493.654105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2493.714768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2493.764910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2493.812121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2493.890028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2493.927982] LNet: 126159:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2493.965700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2494.031970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2494.082922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2494.145696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2494.227882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2494.267804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2495.717644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2496.205123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2496.293816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2496.411666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2496.524517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2496.600865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2496.684259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2496.779706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2496.851650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2496.939541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2496.967894] LNet: 126278:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2497.009180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2497.129818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2497.209360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2497.273418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2497.391899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2497.522899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2498.894555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.137@tcp [ 2499.299459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2499.384836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2499.434026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2499.512213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2499.578035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2499.646607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2499.734978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2499.797729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2499.875950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2499.939256] LNet: 126398:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2499.990670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2500.125907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2500.202592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2500.278160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2500.337866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2500.379541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2501.411824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2501.526377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2501.538347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2501.551154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2501.564222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2501.581660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2501.597806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2501.617867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2501.634928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2501.652382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2501.668071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2501.684624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2501.702786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2501.717568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2501.734706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2501.756348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2508.860556] LNet: 126598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.863801] LNetError: 126598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.919220] LNet: Removed LNI 192.168.202.37@tcp [ 2509.921134] LNetError: 126598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.929748] LNet: Removed LNI 192.168.202.37@tcp1 [ 2510.242818] Key type .llcrypt unregistered [ 2510.244354] Key type ._llcrypt unregistered [ 2528.573305] Key type ._llcrypt registered [ 2528.575341] Key type .llcrypt registered [ 2529.989472] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 20:15:42 (1790900142) [ 2530.599329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.623419] alg: No test for adler32 (adler32-zlib) [ 2531.583524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2531.637168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2531.660894] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2532.418966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2532.452590] LNet: Added LNI 192.168.202.37@tcp2 [8/256/0/180] [ 2533.366689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.37@tcp [ 2533.436171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2534.495200] LNet: Removed LNI 192.168.202.37@tcp [ 2541.157386] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 20:15:54 (1790900154) [ 2542.539951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2544.524520] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:15:57 (1790900157) [ 2553.305343] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:16:05 (1790900165) [ 2560.544536] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:16:13 (1790900173) [ 2568.949876] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:16:22 (1790900182) [ 2577.211450] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:16:30 (1790900190) [ 2584.857357] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:16:38 (1790900198) [ 2590.786714] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:16:44 (1790900204) [ 2598.056550] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:16:51 (1790900211) [ 2608.117323] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:17:00 (1790900220) [ 2617.175731] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:17:09 (1790900229) [ 2624.475384] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:17:17 (1790900237) [ 2631.389672] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:17:24 (1790900244) [ 2638.749760] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:17:31 (1790900251) [ 2646.001509] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:17:39 (1790900259) [ 2653.478808] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:17:46 (1790900266) [ 2661.133990] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:17:54 (1790900274) [ 2669.146519] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:18:02 (1790900282) [ 2676.156355] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:18:09 (1790900289) [ 2676.271534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2676.313888] LNet: 138284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.319758] LNetError: 138284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2676.338178] LNet: Removed LNI 192.168.202.37@tcp2 [ 2676.355176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2676.485883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2676.507688] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2677.541157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2677.563024] LNet: 138433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.568391] LNetError: 138433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.591880] LNet: Removed LNI 192.168.202.37@tcp [ 2677.608995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2677.697264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2677.728586] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2683.252289] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:18:16 (1790900296) [ 2690.886950] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:18:23 (1790900303) [ 2699.083796] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:18:32 (1790900312) [ 2706.859433] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:18:40 (1790900320) [ 2713.757450] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:18:46 (1790900326) [ 2720.603754] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:18:53 (1790900333) [ 2727.932746] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:19:00 (1790900340) [ 2737.152901] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:19:09 (1790900349) [ 2746.040545] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:19:19 (1790900359) [ 2753.119890] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:19:26 (1790900366) [ 2760.878094] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:19:34 (1790900374) [ 2767.884074] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:19:41 (1790900381) [ 2774.713177] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2518 sec ========== 20:19:47 (1790900387) [ 2775.029915] LNet: 145900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.035327] LNetError: 145900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.049401] LNet: Removed LNI 192.168.202.37@tcp [ 2775.726650] Key type .llcrypt unregistered [ 2775.728715] Key type ._llcrypt unregistered [ 2787.725599] Key type ._llcrypt registered [ 2787.727012] Key type .llcrypt registered [ 2788.392316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2788.412496] alg: No test for adler32 (adler32-zlib) [ 2789.542963] Lustre: Lustre: Build Version: 2.17.59_2_g8e10c0d [ 2789.987302] LNet: Added LNI 192.168.202.37@tcp [8/256/0/180] [ 2791.663291] Key type lgssc registered [ 2793.045247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2856.849980] Lustre: Mounted lustre-client - version 2.17.59_2_g8e10c0d [ 2861.211082] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2869.718466] Lustre: Unmounted lustre-client [ 2899.261543] Key type lgssc unregistered [ 2899.549472] LNet: 148874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2899.560533] LNetError: 148874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.595085] LNet: Removed LNI 192.168.202.37@tcp [ 2900.272225] Key type .llcrypt unregistered [ 2900.275503] Key type ._llcrypt unregistered