[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 611309764 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003080] x2apic enabled [ 0.005006] Switched APIC routing to physical x2apic. [ 0.006024] kvm-guest: setup PV IPIs [ 0.009513] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010033] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011020] pid_max: default: 32768 minimum: 301 [ 0.012169] LSM: Security Framework initializing [ 0.013077] Yama: becoming mindful. [ 0.014061] SELinux: Initializing. [ 0.015090] *** VALIDATE selinux *** [ 0.025317] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031310] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033142] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034144] *** VALIDATE tmpfs *** [ 0.035550] *** VALIDATE proc *** [ 0.036401] *** VALIDATE cgroup *** [ 0.038012] *** VALIDATE cgroup2 *** [ 0.039311] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042038] Spectre V2 : User space: Vulnerable [ 0.043013] Speculative Store Bypass: Vulnerable [ 0.046527] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049771] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050033] ... version: 2 [ 0.051029] ... bit width: 48 [ 0.052023] ... generic registers: 4 [ 0.053022] ... value mask: 0000ffffffffffff [ 0.054025] ... max period: 00007fffffffffff [ 0.055024] ... fixed-purpose events: 3 [ 0.056021] ... event mask: 000000070000000f [ 0.057380] rcu: Hierarchical SRCU implementation. [ 0.060000] smp: Bringing up secondary CPUs ... [ 0.061804] x86: Booting SMP configuration: [ 0.062040] .... node #0, CPUs: #1 #2 #3 [ 0.074155] smp: Brought up 1 node, 4 CPUs [ 0.076028] smpboot: Max logical packages: 1 [ 0.077029] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.163947] node 0 deferred pages initialised in 84ms [ 0.168550] devtmpfs: initialized [ 0.173228] x86/mm: Memory block size: 128MB [ 0.182492] gcov: version magic: 0x41383552 [ 0.188392] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.197142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.208245] pinctrl core: initialized pinctrl subsystem [ 0.216307] [ 0.216889] ************************************************************* [ 0.223019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229017] ** ** [ 0.233016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.239025] ** ** [ 0.243019] ** This means that this kernel is built to expose internal ** [ 0.248034] ** IOMMU data structures, which may compromise security on ** [ 0.252019] ** your system. ** [ 0.257262] ** ** [ 0.260018] ** If you see this message and you are not debugging the ** [ 0.262021] ** kernel, report this immediately to your vendor! ** [ 0.267022] ** ** [ 0.269022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273025] ************************************************************* [ 0.277384] NET: Registered protocol family 16 [ 0.279703] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286239] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.289063] cpuidle: using governor menu [ 0.292107] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296059] PCI: Using configuration type 1 for base access [ 0.299144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.310033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.312065] cryptd: max_cpu_qlen set to 1000 [ 0.315264] ACPI: Added _OSI(Module Device) [ 0.318022] ACPI: Added _OSI(Processor Device) [ 0.320020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329623] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340241] ACPI: Interpreter enabled [ 0.342086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.345020] ACPI: Using IOAPIC for interrupt routing [ 0.350205] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354580] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.369000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.370049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.372028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.377101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.383795] acpiphp: Slot [2] registered [ 0.386270] acpiphp: Slot [5] registered [ 0.388221] acpiphp: Slot [6] registered [ 0.390185] acpiphp: Slot [3] registered [ 0.392156] acpiphp: Slot [4] registered [ 0.393240] acpiphp: Slot [7] registered [ 0.395513] acpiphp: Slot [8] registered [ 0.399176] acpiphp: Slot [9] registered [ 0.402147] acpiphp: Slot [10] registered [ 0.405141] acpiphp: Slot [11] registered [ 0.410148] acpiphp: Slot [12] registered [ 0.413154] acpiphp: Slot [13] registered [ 0.419186] acpiphp: Slot [14] registered [ 0.423157] acpiphp: Slot [15] registered [ 0.427164] acpiphp: Slot [16] registered [ 0.433271] acpiphp: Slot [17] registered [ 0.437366] acpiphp: Slot [18] registered [ 0.441710] acpiphp: Slot [19] registered [ 0.445179] acpiphp: Slot [20] registered [ 0.449163] acpiphp: Slot [21] registered [ 0.453181] acpiphp: Slot [22] registered [ 0.455564] acpiphp: Slot [23] registered [ 0.459195] acpiphp: Slot [24] registered [ 0.464895] acpiphp: Slot [25] registered [ 0.468437] acpiphp: Slot [26] registered [ 0.471343] acpiphp: Slot [27] registered [ 0.474526] acpiphp: Slot [28] registered [ 0.477153] acpiphp: Slot [29] registered [ 0.479169] acpiphp: Slot [30] registered [ 0.481142] acpiphp: Slot [31] registered [ 0.484282] PCI host bridge to bus 0000:00 [ 0.487030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.490182] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.493028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.499036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.504032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.511041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.512191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.517787] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.526793] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.536018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.540063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.543023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.545030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.548025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.552008] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.554907] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.559053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.563358] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.569027] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.580851] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.584000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.590000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.599027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.610017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.638022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.664041] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.670032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.682021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.700022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.721000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728748] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.734453] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.741299] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.754490] iommu: Default domain type: Passthrough [ 0.758479] SCSI subsystem initialized [ 0.761907] ACPI: bus type USB registered [ 0.768143] usbcore: registered new interface driver usbfs [ 0.769984] usbcore: registered new interface driver hub [ 0.777445] usbcore: registered new device driver usb [ 0.783000] pps_core: LinuxPPS API ver. 1 registered [ 0.787016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.796070] PTP clock support registered [ 0.801057] EDAC MC: Ver: 3.0.0 [ 0.803177] PCI: Using ACPI for IRQ routing [ 0.806835] NetLabel: Initializing [ 0.809028] NetLabel: domain hash size = 128 [ 0.811016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.817070] NetLabel: unlabeled traffic allowed by default [ 0.823112] vgaarb: loaded [ 0.824572] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.831711] clocksource: Switched to clocksource kvm-clock [ 0.974384] VFS: Disk quotas dquot_6.6.0 [ 0.976131] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.978916] *** VALIDATE ramfs *** [ 0.980218] *** VALIDATE hugetlbfs *** [ 0.981441] pnp: PnP ACPI init [ 0.983705] pnp: PnP ACPI: found 6 devices [ 1.002628] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.006898] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.009349] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.011917] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.015662] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.018527] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.024175] NET: Registered protocol family 2 [ 1.027667] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.034134] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.039350] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.047271] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.052572] TCP: Hash tables configured (established 65536 bind 65536) [ 1.057621] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.062524] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.066967] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.070858] NET: Registered protocol family 1 [ 1.074483] RPC: Registered named UNIX socket transport module. [ 1.078715] RPC: Registered udp transport module. [ 1.081418] RPC: Registered tcp transport module. [ 1.084131] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.087721] NET: Registered protocol family 44 [ 1.089926] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.093745] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.096029] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.098347] PCI: CLS 0 bytes, default 64 [ 1.100705] Unpacking initramfs... [ 3.671189] debug: unmapping init [mem 0xffffa0757cc64000-0xffffa0757ffcffff] [ 3.677942] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.682072] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.686868] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.512511] Initialise system trusted keyrings [ 4.514357] Key type blacklist registered [ 4.520729] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.571053] zbud: loaded [ 4.579308] *** VALIDATE nfs *** [ 4.586557] *** VALIDATE nfs4 *** [ 4.588868] pstore: using deflate compression [ 4.595919] Platform Keyring initialized [ 4.734544] NET: Registered protocol family 38 [ 4.736818] Key type asymmetric registered [ 4.738439] Asymmetric key parser 'x509' registered [ 4.741763] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.746596] io scheduler mq-deadline registered [ 4.748811] io scheduler kyber registered [ 4.751361] io scheduler bfq registered [ 4.754518] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.758275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.761822] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.765306] ACPI: Power Button [PWRF] [ 4.776048] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.786542] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.812424] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.855810] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.892292] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.898354] Non-volatile memory driver v1.3 [ 4.899777] Linux agpgart interface v0.103 [ 4.966990] virtio_blk virtio1: [vda] 145840 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.972847] vda: detected capacity change from 0 to 74670080 [ 5.004255] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.010462] vdb: detected capacity change from 0 to 1073741824 [ 5.027405] libphy: Fixed MDIO Bus: probed [ 5.036439] usbcore: registered new interface driver usbserial_generic [ 5.040428] usbserial: USB Serial support registered for generic [ 5.044126] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.050214] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.052681] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.055901] mousedev: PS/2 mouse device common for all mice [ 5.060829] rtc_cmos 00:05: RTC can wake from S4 [ 5.064361] rtc_cmos 00:05: registered as rtc0 [ 5.071196] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.072457] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.075347] intel_pstate: CPU model not supported [ 5.083081] hid: raw HID events driver (C) Jiri Kosina [ 5.088533] usbcore: registered new interface driver usbhid [ 5.090691] usbhid: USB HID core driver [ 5.094862] drop_monitor: Initializing network drop monitor service [ 5.098202] Initializing XFRM netlink socket [ 5.101330] NET: Registered protocol family 10 [ 5.108348] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.108429] Segment Routing with IPv6 [ 5.114854] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.118589] NET: Registered protocol family 17 [ 5.120260] mpls_gso: MPLS GSO support [ 5.140540] RAS: Correctable Errors collector initialized. [ 5.147154] AVX version of gcm_enc/dec engaged. [ 5.149545] AES CTR mode by8 optimization enabled [ 5.278527] sched_clock: Marking stable (5278248922, 0)->(6652142938, -1373894016) [ 5.290764] registered taskstats version 1 [ 5.299429] Loading compiled-in X.509 certificates [ 5.314067] zswap: loaded using pool lzo/zbud [ 5.413781] Key type big_key registered [ 5.453851] Key type encrypted registered [ 5.459513] ima: No TPM chip found, activating TPM-bypass! [ 5.462879] ima: Allocated hash algorithm: sha1 [ 5.465889] ima: No architecture policies found [ 5.467964] evm: Initialising EVM extended attributes: [ 5.471579] evm: security.selinux [ 5.473652] evm: security.ima [ 5.475539] evm: security.capability [ 5.477612] evm: HMAC attrs: 0x1 [ 5.481605] rtc_cmos 00:05: setting system clock to 2026-08-10 13:54:57 UTC (1786370097) [ 5.491390] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 5.496692] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 5.504380] Write protecting the kernel read-only data: 28672k [ 5.513550] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 5.521664] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 5.596707] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.609640] systemd[1]: Detected virtualization kvm. [ 5.612375] systemd[1]: Detected architecture x86-64. [ 5.614919] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.657691] systemd[1]: No hostname configured. [ 5.660481] systemd[1]: Set hostname to . [ 5.663813] random: systemd: uninitialized urandom read (16 bytes read) [ 5.669204] systemd[1]: Initializing machine ID from random generator. [ 5.751769] random: ln: uninitialized urandom read (6 bytes read) [ 5.966619] random: systemd: uninitialized urandom read (16 bytes read) [ 5.974751] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.985212] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.991725] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.920372] device-mapper: uevent: version 1.0.3 [ 7.924037] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.401431] virtio_net virtio0 ens2: renamed from eth0 [ 9.512763] scsi host0: ata_piix [ 9.552282] scsi host1: ata_piix [ 9.554414] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.559657] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.020974] random: crng init done [ 15.029317] random: 7 urandom warning(s) missed due to ratelimiting [ 15.935347] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.214391] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.702515] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.384573] SELinux: Disabled at runtime. [ 20.486122] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.510500] systemd[1]: Detected virtualization kvm. [ 20.517778] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.725841] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.736136] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.748872] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.759265] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.765759] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.776238] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.789741] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 22.338995] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.306100] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.105727] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.244962] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.856322] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.901248] EDAC sbridge: Ver: 1.1.2 [ 27.995968] Key type dns_resolver registered [ 28.428930] NFS: Registering the id_resolver key type [ 28.435320] Key type id_resolver registered [ 28.437112] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg437-client login: [ 87.104618] libcfs: loading out-of-tree module taints kernel. [ 87.249031] Key type ._llcrypt registered [ 87.251688] Key type .llcrypt registered [ 87.567199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 87.575956] alg: No test for adler32 (adler32-zlib) [ 88.864642] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 89.410066] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 91.175200] Key type lgssc registered [ 92.787816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 193.994029] hrtimer: interrupt took 7897829 ns [ 208.200160] Lustre: Mounted lustre-client - version 2.17.56_50_g6d2b241 [ 214.141479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.354523] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing check_logdir /tmp/testlogs/ [ 233.951573] Lustre: lustre-OST0000-osc-ffffa075d2658800: disconnect after 23s idle [ 234.302689] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing yml_node [ 238.778985] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 241.306966] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 243.183707] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 244.992659] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 10 09:58:55 EDT 2026 [ 258.871675] Lustre: Unmounted lustre-client [ 278.444376] Key type lgssc unregistered [ 278.747959] LNet: 8415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.759180] LNetError: 8415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.774409] LNet: Removed LNI 192.168.204.37@tcp [ 279.364183] Key type .llcrypt unregistered [ 279.367310] Key type ._llcrypt unregistered [ 298.646691] Key type ._llcrypt registered [ 298.649381] Key type .llcrypt registered [ 298.825747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 298.848845] alg: No test for adler32 (adler32-zlib) [ 299.706851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 299.763153] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 299.788789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 302.926605] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 303.014151] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.021859] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 303.037697] LNet: Removed LNI 192.168.204.37@tcp [ 303.581464] Key type .llcrypt unregistered [ 303.583705] Key type ._llcrypt unregistered [ 308.254774] Key type ._llcrypt registered [ 308.257863] Key type .llcrypt registered [ 309.184810] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:00:00 (1786370400) [ 309.458772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 309.470665] alg: No test for adler32 (adler32-zlib) [ 310.346573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.417496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.428512] LNet: 9715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.443527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.477581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 310.493040] LNet: 9722:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 315.391642] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:00:06 (1786370406) [ 315.504588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.524228] LNet: 10297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.545706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.600640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 320.659787] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:00:11 (1786370411) [ 320.805752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.822346] LNet: 10880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.850513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.948560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 326.536340] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:00:17 (1786370417) [ 326.622211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.657125] LNet: 11459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.707601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.807490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 333.214929] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:00:23 (1786370423) [ 333.373655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.406649] LNet: 12051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.438391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.503183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 333.811575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 339.018636] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:00:29 (1786370429) [ 339.116707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.167846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.264585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.539419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.871901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 345.455856] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:00:36 (1786370436) [ 345.664639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.680636] LNet: 13240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.684880] LNet: 13240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 345.701682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.768754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 351.913554] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:00:42 (1786370442) [ 352.145534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.172319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.224766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 352.252292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.516598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 352.553692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.837515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 352.885486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.172097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 353.215847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 353.460169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 353.486661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.732845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 353.778063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.076513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 354.104920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.368721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 354.399394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.645355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 354.677465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 355.019375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 355.044895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.327668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 355.356871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.663962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 355.692037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 355.951669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 355.978555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 361.788499] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:00:52 (1786370452) [ 362.088611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.111633] LNet: 14507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.116839] LNet: 14507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 362.135163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.185366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 362.209620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 367.447852] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:00:58 (1786370458) [ 367.583385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.619650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.673746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 367.704179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 373.158421] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:01:04 (1786370464) [ 373.404668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.438589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.524316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 373.558802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 378.292707] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:01:09 (1786370469) [ 378.471756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.504723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.554467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 378.580208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 382.836650] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:01:14 (1786370474) [ 383.114795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.168593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.241721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 383.272210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 389.129781] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:01:19 (1786370479) [ 389.233314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.291390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.397310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 389.433338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 394.298240] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:01:25 (1786370485) [ 394.428413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.447302] LNet: 18037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.453375] LNet: 18037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 394.472233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.547709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 394.878803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 400.016443] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:01:30 (1786370490) [ 400.180447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.205284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.426517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.608993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 406.130816] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:01:37 (1786370497) [ 406.253709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.305713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.420364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.605327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 411.548703] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:01:42 (1786370502) [ 411.637845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.678979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.017338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.258838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.686233] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:01:47 (1786370507) [ 416.891750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.923173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.990415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.045727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.780821] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:01:52 (1786370512) [ 421.881217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.917853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.975249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.015301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 426.498879] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:01:57 (1786370517) [ 426.621469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.668164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.713132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 431.508493] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:02:02 (1786370522) [ 431.597649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.630985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.717993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 436.819390] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:02:07 (1786370527) [ 436.961173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.011953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.062150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 437.400189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 442.643802] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:02:13 (1786370533) [ 442.738937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.767413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.827417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 442.857895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 448.126418] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:02:19 (1786370539) [ 448.274513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.334193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.448793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 448.497051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 454.644863] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:02:25 (1786370545) [ 454.836955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.859383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.915658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 454.957610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 460.797910] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:02:31 (1786370551) [ 460.918771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.937302] LNet: 25234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.943637] LNet: 25234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 460.970688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.028479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 461.062589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 466.631527] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:02:37 (1786370557) [ 466.941908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.009578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.102902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 467.165359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 467.192597] LNetError: 25833:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 467.239529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 467.271198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 472.824500] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:02:43 (1786370563) [ 473.008689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.130700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.296515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GO1OeFXruM [ 473.388633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GO1OeFXruM [ 473.456632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GO1OeFXruM [ 478.932615] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:02:49 (1786370569) [ 479.262949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.307839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.365400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 479.398351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 479.453646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 485.609707] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:02:56 (1786370576) [ 485.923763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.961180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.023657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 486.044089] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 487.438779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 488.819887] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 490.751235] Key type lgssc registered [ 491.859691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 495.125260] LNetError: 27594:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.37@tcp99 has no available nets [ 497.683628] Key type lgssc unregistered [ 498.080984] LNetError: 28710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 498.096420] LNet: Removed LNI 192.168.204.37@tcp [ 498.840374] Key type .llcrypt unregistered [ 498.842855] Key type ._llcrypt unregistered [ 505.188498] Key type ._llcrypt registered [ 505.191079] Key type .llcrypt registered [ 506.464264] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:03:17 (1786370597) [ 506.934905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 506.957562] alg: No test for adler32 (adler32-zlib) [ 507.823136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.881967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 509.695337] LNet: Message forwarding will be enabled in 2s [ 512.799704] LNet: Message forwarding enabled [ 513.183575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 513.199288] LNet: Message forwarding will stop in 2s to 6s [ 517.710133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 517.963194] LNet: Message forwarding disabled [ 519.200363] LNet: Message forwarding will be enabled in 2s [ 523.103765] LNet: Message forwarding enabled [ 526.544919] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:03:37 (1786370617) [ 526.866613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.921565] LNet: 30118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.870540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.926958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 534.777449] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:03:44 (1786370624) [ 535.164792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.186771] LNet: 30699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.215788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.275672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 535.325393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 535.390438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 535.429433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 535.456285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 535.485200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 535.529572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 535.557922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 535.592598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 535.648322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 535.715696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 535.760433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 535.787916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 535.838274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 535.882826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 535.919910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 535.970361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 535.999720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 536.055945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 536.073445] LNetError: 30742:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 541.819931] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:03:52 (1786370632) [ 542.120930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.145972] LNet: 31313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.160679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.210261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.372523] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:03:58 (1786370638) [ 549.044958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg437-server [ 550.839460] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:04:01 (1786370641) [ 550.940362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.986023] LNet: 32240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.049208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.113958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 551.157710] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 551.876968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[38-42]@tcp [ 551.892350] LNet: 32377:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 551.930961] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.38@tcp is being used as a gateway but routing feature is not turned on [ 551.949651] LNetError: 32247:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.38@tcp has gone from up to down [ 552.937056] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 552.956650] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 553.955409] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 553.963349] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 556.011890] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.39@tcp is being used as a gateway but routing feature is not turned on [ 556.022379] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 557.598940] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:04:08 (1786370648) [ 559.095427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg437-server [ 560.107758] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.39@tcp is being used as a gateway but routing feature is not turned on [ 560.118661] LNetError: 32247:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 560.612582] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:04:11 (1786370651) [ 560.687295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.714429] LNet: 33307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.721985] LNetError: 33307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.750402] LNet: Removed LNI 192.168.204.37@tcp [ 560.773688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.815140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 560.832162] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 561.403951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[38-43/2]@tcp [ 561.420278] LNet: 33429:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 561.425282] LNet: 33429:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 561.446534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[38-43/2]@tcp [ 561.453842] LNetError: 33316:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.42@tcp has gone from up to down [ 561.487844] LNetError: 33316:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 566.725135] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:04:17 (1786370657) [ 566.857624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.877544] LNetError: 34021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.891371] LNet: Removed LNI 192.168.204.37@tcp [ 566.910469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.999439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 567.060587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.090282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.161074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 567.214770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.249089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.298736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 567.367997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.397146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.451305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 567.512991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.538475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.575183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 567.643336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.672577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.723420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 567.792430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.841633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.882934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 567.947147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.975326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.028339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.091101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.143594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.184724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 568.258718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.299247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.352959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.423480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.469631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.513748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 568.561681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.594831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.658267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.982706] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:04:24 (1786370664) [ 574.308739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg437-server [ 575.574701] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:04:26 (1786370666) [ 577.010051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg437-server [ 578.298660] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:04:29 (1786370669) [ 578.377873] LNet: 35413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.383436] LNet: 35413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 578.830414] Key type .llcrypt unregistered [ 578.832315] Key type ._llcrypt unregistered [ 583.220658] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 583.323576] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 583.325869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 583.336836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 585.746867] Key type ._llcrypt registered [ 585.749714] Key type .llcrypt registered [ 585.898779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 585.919874] alg: No test for adler32 (adler32-zlib) [ 586.824631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.887960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.907529] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 587.464476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 587.477427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.904762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 588.924712] LNet: Removed LNI 10.1.2.3@tcp [ 589.016820] LNet: 36142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.024140] LNetError: 36142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.037376] LNet: Removed LNI 192.168.204.37@tcp [ 589.481421] Key type .llcrypt unregistered [ 589.483410] Key type ._llcrypt unregistered [ 598.088192] Key type ._llcrypt registered [ 598.090896] Key type .llcrypt registered [ 599.164450] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:04:49 (1786370689) [ 599.666679] Key type .llcrypt unregistered [ 599.670474] Key type ._llcrypt unregistered [ 604.250598] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 604.370713] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 604.379559] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 604.385707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 606.829421] Key type ._llcrypt registered [ 606.831047] Key type .llcrypt registered [ 606.973653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.982826] alg: No test for adler32 (adler32-zlib) [ 607.864603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.934304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.961717] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 608.557681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 608.587342] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 610.696147] LNet: 37642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.705438] LNetError: 37642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 610.713133] LNet: Removed LNI 10.1.2.3@tcp [ 610.728732] LNet: Removed LNI 192.168.204.37@tcp [ 611.358327] Key type .llcrypt unregistered [ 611.360498] Key type ._llcrypt unregistered [ 619.488514] Key type ._llcrypt registered [ 619.490630] Key type .llcrypt registered [ 620.479733] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:05:11 (1786370711) [ 621.008257] Key type .llcrypt unregistered [ 621.012863] Key type ._llcrypt unregistered [ 626.054991] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 626.144874] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 626.147376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 626.157847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 628.660969] Key type ._llcrypt registered [ 628.663302] Key type .llcrypt registered [ 628.860478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 628.878442] alg: No test for adler32 (adler32-zlib) [ 629.764082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.831575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 629.862135] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 631.322292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 631.343711] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 632.874298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 632.898392] LNet: Removed LNI 10.1.2.3@tcp [ 632.914903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 632.943258] LNet: Removed LNI 10.1.2.31@tcp [ 632.955530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 632.981416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 633.650677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 633.662092] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 634.341570] LNet: 39570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.348844] LNetError: 39570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 634.352870] LNet: Removed LNI 10.1.2.31@tcp [ 634.360937] LNet: Removed LNI 10.1.2.3@tcp [ 634.882502] Key type .llcrypt unregistered [ 634.885508] Key type ._llcrypt unregistered [ 644.786849] Key type ._llcrypt registered [ 644.789545] Key type .llcrypt registered [ 645.880164] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:05:36 (1786370736) [ 646.240498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.256557] alg: No test for adler32 (adler32-zlib) [ 647.196713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.241747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.267918] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 647.882452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.902706] LNet: 40527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.907774] LNetError: 40527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.923956] LNet: Removed LNI 192.168.204.37@tcp [ 647.946832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.016678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.37@tcp [ 648.055810] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 648.693948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.37@tcp [ 648.724875] LNet: Removed LNI 192.168.204.37@tcp [ 648.792660] LNet: 40662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.391507] Key type .llcrypt unregistered [ 649.395550] Key type ._llcrypt unregistered [ 658.503851] Key type ._llcrypt registered [ 658.508513] Key type .llcrypt registered [ 659.667492] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:05:50 (1786370750) [ 660.905179] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg437-server [ 662.107899] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:05:53 (1786370753) [ 662.601413] Key type .llcrypt unregistered [ 662.604863] Key type ._llcrypt unregistered [ 666.702075] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.837652] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.846170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.850338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.311361] Key type ._llcrypt registered [ 669.314352] Key type .llcrypt registered [ 669.469478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.482597] alg: No test for adler32 (adler32-zlib) [ 670.367750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.421723] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 671.033509] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 672.941594] LNet: 42504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.946610] LNetError: 42504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.951058] LNet: Removed LNI 192.168.204.37@tcp1 [ 672.958889] LNet: Removed LNI 10.1.2.3@tcp2 [ 673.418644] Key type .llcrypt unregistered [ 673.423096] Key type ._llcrypt unregistered [ 683.407200] Key type ._llcrypt registered [ 683.408791] Key type .llcrypt registered [ 684.360755] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:06:15 (1786370775) [ 684.666994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 684.678457] alg: No test for adler32 (adler32-zlib) [ 685.561065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.600448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.615383] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 686.203736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.224666] LNet: 43480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.232905] LNetError: 43480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.245811] LNet: Removed LNI 192.168.204.37@tcp [ 686.261838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.325492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 686.353880] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 691.680873] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:06:22 (1786370782) [ 691.815420] LNet: 44068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.824255] LNetError: 44068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.861295] LNet: Removed LNI 192.168.204.37@tcp [ 692.428704] Key type .llcrypt unregistered [ 692.431814] Key type ._llcrypt unregistered [ 697.374572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 697.497471] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 697.513424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 697.523761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.904181] Key type ._llcrypt registered [ 699.906235] Key type .llcrypt registered [ 700.080753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 700.109804] alg: No test for adler32 (adler32-zlib) [ 700.972679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.024918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.042915] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 701.604862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 701.623866] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.618708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.635622] LNet: 44818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.642684] LNetError: 44818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 703.650188] LNet: Removed LNI 10.1.2.3@tcp [ 703.658450] LNet: Removed LNI 192.168.204.37@tcp [ 703.678418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.723891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 703.746974] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 703.750729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.073093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.095239] LNet: Removed LNI 10.1.2.3@tcp [ 704.116874] LNet: Removed LNI 192.168.204.37@tcp [ 704.140885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.192961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 704.212950] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 704.216762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.524372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 704.541780] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 705.427157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.441466] LNet: 45007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.444989] LNet: 45007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 705.450616] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 705.455155] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 705.459223] LNet: Removed LNI 10.1.2.3@tcp [ 705.460810] LNet: Removed LNI 192.168.204.37@tcp [ 705.473761] LNet: Removed LNI 192.168.204.37@tcp2 [ 705.499701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.570720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 705.590756] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 705.596655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 705.600814] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 710.649872] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:06:41 (1786370801) [ 710.815340] LNet: 45604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.819936] LNetError: 45604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 710.824267] LNetError: 45604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 710.829585] LNet: Removed LNI 10.1.2.3@tcp [ 710.835633] LNet: Removed LNI 192.168.204.37@tcp [ 710.845605] LNet: Removed LNI 192.168.204.37@tcp2 [ 711.294836] Key type .llcrypt unregistered [ 711.296755] Key type ._llcrypt unregistered [ 715.695666] Key type ._llcrypt registered [ 715.701599] Key type .llcrypt registered [ 716.080735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.118577] alg: No test for adler32 (adler32-zlib) [ 717.037574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.088482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.118703] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 717.759425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.780292] LNet: 46003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.785518] LNetError: 46003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.800298] LNet: Removed LNI 192.168.204.37@tcp [ 717.820817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.881338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 717.906231] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 718.325457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.355405] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.360839] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.378950] LNet: Removed LNI 192.168.204.37@tcp [ 718.403955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.455806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 718.477225] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 718.848988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.871657] LNet: Removed LNI 192.168.204.37@tcp [ 718.896584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.943662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 718.965331] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 719.241785] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 719.374505] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 719.377938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 719.382967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 721.684762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.701173] LNet: 46141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.707225] LNet: 46141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 721.713706] LNetError: 46141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.720491] LNetError: 46141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 721.732743] LNet: Removed LNI 192.168.204.37@tcp [ 721.749583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.796312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 721.815463] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 727.064827] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:06:58 (1786370818) [ 727.332577] LNet: 46738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.345820] LNetError: 46738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.368655] LNet: Removed LNI 192.168.204.37@tcp [ 727.902291] Key type .llcrypt unregistered [ 727.904771] Key type ._llcrypt unregistered [ 732.420317] Key type ._llcrypt registered [ 732.421920] Key type .llcrypt registered [ 732.585975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 732.601296] alg: No test for adler32 (adler32-zlib) [ 733.464843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.507592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.524987] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 734.290294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.313765] LNet: 47141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.322601] LNetError: 47141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.343368] LNet: Removed LNI 192.168.204.37@tcp [ 734.379369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.452558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 734.477127] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 739.730186] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:07:10 (1786370830) [ 739.824107] LNet: 47725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.834619] LNetError: 47725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.864190] LNet: Removed LNI 192.168.204.37@tcp [ 740.414851] Key type .llcrypt unregistered [ 740.416539] Key type ._llcrypt unregistered [ 744.949385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 745.025451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 745.027437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 745.029419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 747.426407] Key type ._llcrypt registered [ 747.429981] Key type .llcrypt registered [ 747.564469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.576933] alg: No test for adler32 (adler32-zlib) [ 748.453249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.504136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.524809] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 749.126375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.144689] LNet: 48185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.149310] LNetError: 48185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.160883] LNet: Removed LNI 192.168.204.37@tcp [ 749.176865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.233578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 749.253711] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 753.861206] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:07:25 (1786370845) [ 754.002825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.016303] LNet: 48783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.024488] LNetError: 48783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.041209] LNet: Removed LNI 192.168.204.37@tcp [ 754.060427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.139474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 758.637315] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:07:29 (1786370849) [ 758.766424] LNet: 49356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.490187] Key type .llcrypt unregistered [ 759.493490] Key type ._llcrypt unregistered [ 764.198225] Key type ._llcrypt registered [ 764.201748] Key type .llcrypt registered [ 764.325136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.339574] alg: No test for adler32 (adler32-zlib) [ 765.170264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.202562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.220771] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 765.732717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.745568] LNet: 49766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.751446] LNetError: 49766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.763083] LNet: Removed LNI 192.168.204.37@tcp [ 765.777460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.823690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 765.844705] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 770.342651] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:07:41 (1786370861) [ 770.478588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.504900] LNet: 50359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.508657] LNetError: 50359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.523628] LNet: Removed LNI 192.168.204.37@tcp [ 770.541263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.595753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 774.921413] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:07:46 (1786370866) [ 776.115523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 777.391244] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:07:48 (1786370868) [ 778.380977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 779.566560] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:07:50 (1786370870) [ 780.576282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 781.667517] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:07:52 (1786370872) [ 782.704743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 783.939333] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:07:54 (1786370874) [ 784.990574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 786.326141] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:07:57 (1786370877) [ 786.487972] LNet: 52409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.937311] Key type .llcrypt unregistered [ 786.940462] Key type ._llcrypt unregistered [ 791.276456] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.466179] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.470970] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.474433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.881470] Key type ._llcrypt registered [ 793.884177] Key type .llcrypt registered [ 794.028338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.043066] alg: No test for adler32 (adler32-zlib) [ 794.918938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.982258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 795.010030] LNet: Added LNI 192.168.204.37@tcp [8/512/0/180] [ 795.014077] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 795.016884] LNet: 52736:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 795.092927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.108329] LNet: 52755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.113680] LNetError: 52755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.118689] LNet: Removed LNI 10.1.2.3@tcp [ 795.120850] LNetError: 52755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.136786] LNet: Removed LNI 192.168.204.37@tcp [ 795.152675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.197330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 795.223217] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 795.261601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.278495] LNet: 52779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.285317] LNetError: 52779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.298395] LNet: Removed LNI 192.168.204.37@tcp [ 795.317080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.366492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 795.384352] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 795.388795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.391299] LNet: 52787:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 795.457345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.474851] LNet: 52805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.478935] LNetError: 52805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.483102] LNet: Removed LNI 10.1.2.3@tcp [ 795.486044] LNetError: 52805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.498354] LNet: Removed LNI 192.168.204.37@tcp [ 795.516828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.568969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 795.587499] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 795.591411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.594061] LNet: 52813:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 795.711079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.730070] LNet: 52839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.739852] LNetError: 52839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.745926] LNet: Removed LNI 10.1.2.3@tcp [ 795.750742] LNetError: 52839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.768560] LNet: Removed LNI 192.168.204.37@tcp [ 795.789480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.834969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 795.854051] LNet: Added LNI 192.168.204.37@tcp [8/256/0/182] [ 795.858058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 795.924753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.942251] LNet: 52867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.945809] LNetError: 52867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.951724] LNet: Removed LNI 10.1.2.3@tcp [ 795.954012] LNetError: 52867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.966127] LNet: Removed LNI 192.168.204.37@tcp [ 795.988543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.054884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 796.079156] LNet: Added LNI 192.168.204.37@tcp [8/256/0/182] [ 796.083798] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 796.086151] LNet: 52875:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 796.166201] LNet: Removed LNI 10.1.2.3@tcp [ 796.178883] LNet: Removed LNI 192.168.204.37@tcp [ 796.686446] Key type .llcrypt unregistered [ 796.689613] Key type ._llcrypt unregistered [ 804.724823] Key type ._llcrypt registered [ 804.726921] Key type .llcrypt registered [ 805.671607] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:08:16 (1786370896) [ 806.829322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 808.407220] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:08:19 (1786370899) [ 808.953285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 808.966495] alg: No test for adler32 (adler32-zlib) [ 809.834694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.892725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.911784] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 810.595886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.37 [ 810.618108] LNet: 54143:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 810.624665] LNet: Removed LNI 192.168.204.37@tcp [ 815.690141] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:08:26 (1786370906) [ 815.925862] LNet: 54713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.931308] LNetError: 54713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.945769] LNet: Removed LNI 192.168.204.37@tcp [ 816.430180] Key type .llcrypt unregistered [ 816.434239] Key type ._llcrypt unregistered [ 821.014242] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.211344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.731408] Key type ._llcrypt registered [ 823.733423] Key type .llcrypt registered [ 823.843056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.855732] alg: No test for adler32 (adler32-zlib) [ 824.958768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.162112] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:08:40 (1786370920) [ 830.225639] LNet: 55621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.232278] LNetError: 55621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.255294] LNet: Removed LNI 10.1.2.3@tcp [ 830.791543] Key type .llcrypt unregistered [ 830.793498] Key type ._llcrypt unregistered [ 835.557879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.753984] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.756907] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.761371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 838.447306] Key type ._llcrypt registered [ 838.449360] Key type .llcrypt registered [ 838.583885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.593880] alg: No test for adler32 (adler32-zlib) [ 839.527907] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.666338] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:08:56 (1786370936) [ 846.960347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 848.635654] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:08:59 (1786370939) [ 848.883769] LNet: 56819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.890094] LNetError: 56819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 848.904482] LNet: Removed LNI 10.1.2.3@tcp [ 849.495713] Key type .llcrypt unregistered [ 849.502544] Key type ._llcrypt unregistered [ 855.125269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.473631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.476982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.483924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.048676] Key type ._llcrypt registered [ 857.054386] Key type .llcrypt registered [ 857.222356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.252282] alg: No test for adler32 (adler32-zlib) [ 858.179958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.249639] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 866.042325] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:09:16 (1786370956) [ 866.483882] LNet: 58043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.489965] LNetError: 58043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 866.502092] LNet: Removed LNI 10.1.2.3@tcp [ 866.984164] Key type .llcrypt unregistered [ 866.986362] Key type ._llcrypt unregistered [ 871.434596] Key type ._llcrypt registered [ 871.437714] Key type .llcrypt registered [ 871.630925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.659470] alg: No test for adler32 (adler32-zlib) [ 872.600789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 872.640922] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 878.764268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 879.332753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_interrupt -r 1 [ 879.367835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_interrupt -r 1 [ 879.437679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 879.462944] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 879.671915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 879.701904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 879.722592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 879.751815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 879.775439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 879.798701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 879.830592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 879.844114] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 879.894278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 879.946510] LNet: 58485:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 879.984859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 880.064651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 880.093173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 880.127763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 880.161757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 880.188777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 880.215387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 880.669986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_dropped -r 1 [ 880.700162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_dropped -r 1 [ 880.728345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 880.754386] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 880.941893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 880.985753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 881.012288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 881.043085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 881.070972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 881.103091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 881.129678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 881.155584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 881.171597] LNet: 58594:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 881.188413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 881.219140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 881.248335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 881.280405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 881.309835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.339338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 881.371826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 881.880576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_aborted -r 1 [ 881.918414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_aborted -r 1 [ 881.959826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 881.980900] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 881.990587] LNet: Skipped 1 previous similar message [ 882.230358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 882.255589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 882.281651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 882.308671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 882.334354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 882.363372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 882.393672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 882.422970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 882.437457] LNet: 58704:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 882.453672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 882.481417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 882.532798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 882.602202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 882.668671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.719448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.773629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 883.225427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_no_route -r 1 [ 883.259624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_no_route -r 1 [ 883.298439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 883.513128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 883.546075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 883.573988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 883.602802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 883.629600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 883.657546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 883.681646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 883.704746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 883.732251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 883.760564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 883.788927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 883.810600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 883.837961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 883.865926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 883.897322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 884.422843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_timeout -r 1 [ 884.452386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_timeout -r 1 [ 884.483134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 884.508093] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 884.513501] LNet: Skipped 1 previous similar message [ 884.721781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 884.749653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 884.775740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 884.803775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 884.829498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 884.851445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 884.876598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 884.906817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 884.926658] LNet: 58922:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 884.931986] LNet: 58922:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 884.950732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 884.983569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 885.007465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 885.039418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 885.076742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 885.111817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 885.148758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 885.607273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_error -r 1 [ 885.641968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_error -r 1 [ 885.671253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 885.919818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 885.943759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 885.968973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 886.001686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 886.044481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 886.081778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 886.117874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 886.152562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 886.184338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 886.215372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 886.266095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 886.320162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 886.381943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 886.413878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 886.440573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 886.678630] LNet: 59076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.686609] LNetError: 59076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.704856] LNet: Removed LNI 192.168.204.37@tcp [ 886.995569] Key type .llcrypt unregistered [ 886.997710] Key type ._llcrypt unregistered [ 899.786101] Key type ._llcrypt registered [ 899.789230] Key type .llcrypt registered [ 901.046946] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:09:51 (1786370991) [ 902.110824] Key type .llcrypt unregistered [ 902.115792] Key type ._llcrypt unregistered [ 906.879410] Key type ._llcrypt registered [ 906.880987] Key type .llcrypt registered [ 907.064983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 907.079723] alg: No test for adler32 (adler32-zlib) [ 907.977131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 908.000696] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 913.594902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 922.280664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.319628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 922.345644] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 923.353000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_interrupt -r 1 [ 923.378348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_interrupt -r 1 [ 923.401829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_interrupt -r 1 [ 923.424808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_interrupt -r 1 [ 923.450888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_interrupt -r 1 [ 923.474033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_interrupt -r 1 [ 923.497573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_interrupt -r 1 [ 923.526702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_interrupt -r 1 [ 923.557471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 923.570191] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 923.573744] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 923.705604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 923.730821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 923.763438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 923.785300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 923.810665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 923.829882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 923.850790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 923.891374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 923.912559] LNet: 60550:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 923.932528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 923.958405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 923.984552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 924.013301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 924.037255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 924.063550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 924.097359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 924.620513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_dropped -r 1 [ 924.659463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_dropped -r 1 [ 924.698239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_dropped -r 1 [ 924.736257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_dropped -r 1 [ 924.778727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_dropped -r 1 [ 924.825813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_dropped -r 1 [ 924.858245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_dropped -r 1 [ 924.891533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_dropped -r 1 [ 924.922364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 924.936909] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 924.941647] LNet: Skipped 2 previous similar messages [ 925.149475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 925.179253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 925.216675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 925.241507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 925.263348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 925.288839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 925.311448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 925.334769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 925.348545] LNet: 60671:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 925.361562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 925.388870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 925.414815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 925.440267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 925.462614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 925.488632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 925.513331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 925.957671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_aborted -r 1 [ 925.983234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_aborted -r 1 [ 926.006686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_aborted -r 1 [ 926.029547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_aborted -r 1 [ 926.059370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_aborted -r 1 [ 926.087874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_aborted -r 1 [ 926.120776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_aborted -r 1 [ 926.156989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_aborted -r 1 [ 926.213924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 926.230264] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 926.236196] LNet: Skipped 2 previous similar messages [ 926.391185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.419824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 926.440600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.467033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.495867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.519700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.544266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.565799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.577918] LNet: 60792:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.591954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 926.622574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 926.649050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 926.671073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.695530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.727473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.748654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 927.111236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_no_route -r 1 [ 927.141229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_no_route -r 1 [ 927.181477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_no_route -r 1 [ 927.207630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_no_route -r 1 [ 927.232603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_no_route -r 1 [ 927.265458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_no_route -r 1 [ 927.290258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_no_route -r 1 [ 927.326235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_no_route -r 1 [ 927.351755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 927.537095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 927.558850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 927.577573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 927.595869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 927.614803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 927.635818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 927.661934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 927.682426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 927.707344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 927.739266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 927.761085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 927.785035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 927.802614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 927.829447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.863347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 928.276836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_timeout -r 1 [ 928.308838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_timeout -r 1 [ 928.334884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_timeout -r 1 [ 928.360478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_timeout -r 1 [ 928.386221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_timeout -r 1 [ 928.413849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_timeout -r 1 [ 928.442305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_timeout -r 1 [ 928.467489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_timeout -r 1 [ 928.493030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 928.509853] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 928.516547] LNet: Skipped 5 previous similar messages [ 928.729924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 928.751555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 928.783654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 928.814385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 928.842414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 928.868108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 928.896855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 928.928223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 928.943429] LNet: 61035:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.948591] LNet: 61035:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 928.970596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 928.998788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.016848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 929.041856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 929.067694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 929.091901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.120399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 929.491835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_error -r 1 [ 929.523806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_error -r 1 [ 929.553577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_error -r 1 [ 929.576992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_error -r 1 [ 929.602554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_error -r 1 [ 929.629424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_error -r 1 [ 929.658901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_error -r 1 [ 929.693266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_error -r 1 [ 929.720297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 929.967627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 930.000032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 930.024791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 930.046153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 930.070417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 930.094875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 930.115263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 930.140292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 930.171111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 930.196862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 930.217872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.243498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.281854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.312628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.348135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 933.683386] LNet: 61236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.690566] LNetError: 61236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.698297] LNet: Removed LNI 192.168.204.37@tcp [ 933.710729] LNet: Removed LNI 192.168.204.37@tcp1 [ 934.048728] Key type .llcrypt unregistered [ 934.050251] Key type ._llcrypt unregistered [ 948.375508] Key type ._llcrypt registered [ 948.377338] Key type .llcrypt registered [ 949.545985] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:10:40 (1786371040) [ 950.176858] Key type .llcrypt unregistered [ 950.181614] Key type ._llcrypt unregistered [ 954.392694] Key type ._llcrypt registered [ 954.395759] Key type .llcrypt registered [ 954.562201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.585738] alg: No test for adler32 (adler32-zlib) [ 955.449526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 955.474825] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 961.166840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 961.768972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_dropped -r 1 [ 961.797949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_dropped -r 1 [ 961.830446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 961.842946] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 962.056862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 962.086093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 962.111464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 962.136906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 962.173225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 962.194763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 962.220412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 962.239640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 962.253691] LNet: 62488:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 962.269845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 962.297090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 962.330793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 962.353966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 962.382636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 962.412932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 962.438448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 962.894565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_error -r 1 [ 962.945569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_error -r 1 [ 962.982422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 963.000129] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 963.174886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 963.195763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 963.217461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 963.244171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 963.270837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 963.291878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 963.320819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 963.344774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 963.358844] LNet: 62597:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 963.373941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 963.398118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 963.429805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.453900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.481492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.519795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.546549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 963.936783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_timeout -r 1 [ 963.960416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_timeout -r 1 [ 963.983372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 964.166701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 964.190040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 964.216730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 964.239322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 964.268708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 964.298738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 964.327496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 964.358322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 964.369468] LNet: 62706:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 964.382393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 964.405184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 964.431783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 964.455277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 964.479631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 964.516565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 964.551093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 964.788281] LNet: 62751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.795483] LNetError: 62751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.816576] LNet: Removed LNI 192.168.204.37@tcp [ 965.115789] Key type .llcrypt unregistered [ 965.118754] Key type ._llcrypt unregistered [ 976.133712] Key type ._llcrypt registered [ 976.137798] Key type .llcrypt registered [ 976.910300] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:11:08 (1786371068) [ 977.550673] Key type .llcrypt unregistered [ 977.553148] Key type ._llcrypt unregistered [ 981.840104] Key type ._llcrypt registered [ 981.842132] Key type .llcrypt registered [ 981.992748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.002508] alg: No test for adler32 (adler32-zlib) [ 982.860512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 982.884148] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 988.185531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 995.125358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.174140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 995.194373] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 996.551787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_dropped -r 1 [ 996.580535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e remote_dropped -r 1 [ 996.607322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_dropped -r 1 [ 996.635663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e remote_dropped -r 1 [ 996.665325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e remote_dropped -r 1 [ 996.695933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e remote_dropped -r 1 [ 996.726378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e remote_dropped -r 1 [ 996.769173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e remote_dropped -r 1 [ 996.812375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 996.838292] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 996.847242] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.137@tcp [ 997.012374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.046554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 997.084256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.131373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.180587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.218561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.247971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.285954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.310937] LNet: 64219:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 997.349805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.386654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.420952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.456430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.485267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.528441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 997.569794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 997.609414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 998.122649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_error -r 1 [ 998.160969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e remote_error -r 1 [ 998.206685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_error -r 1 [ 998.252402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e remote_error -r 1 [ 998.293967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e remote_error -r 1 [ 998.334558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e remote_error -r 1 [ 998.382204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e remote_error -r 1 [ 998.421562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e remote_error -r 1 [ 998.460539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 998.481218] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 998.488534] LNet: Skipped 2 previous similar messages [ 998.757536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 998.806530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 998.855552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 998.902545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 998.961312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 998.993276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.025151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.056535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.069834] LNet: 64342:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 999.090215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.171240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.204350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.249791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.281950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.311483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.342671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 999.371497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.044899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_timeout -r 1 [ 1000.108674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e remote_timeout -r 1 [ 1000.167350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_timeout -r 1 [ 1000.217515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e remote_timeout -r 1 [ 1000.259664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e remote_timeout -r 1 [ 1000.301171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e remote_timeout -r 1 [ 1000.341362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e remote_timeout -r 1 [ 1000.387978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e remote_timeout -r 1 [ 1000.424103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.137@tcp [ 1000.445194] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1000.696495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1000.727093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.758340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.791225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.819247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.847341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.878165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.904358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.920309] LNet: 64465:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1000.941566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.967303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.999745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1001.037292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1001.087848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1001.135852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1001.190336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1001.235432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.765133] LNet: 64547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.775153] LNetError: 64547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.785974] LNet: Removed LNI 192.168.204.37@tcp [ 1005.801272] LNet: Removed LNI 192.168.204.37@tcp1 [ 1006.178182] Key type .llcrypt unregistered [ 1006.179762] Key type ._llcrypt unregistered [ 1022.149601] Key type ._llcrypt registered [ 1022.151089] Key type .llcrypt registered [ 1023.280954] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:11:54 (1786371114) [ 1025.300575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1026.903733] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:11:57 (1786371117) [ 1027.817409] Key type .llcrypt unregistered [ 1027.821152] Key type ._llcrypt unregistered [ 1033.868512] Key type ._llcrypt registered [ 1033.870699] Key type .llcrypt registered [ 1034.040891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.056806] alg: No test for adler32 (adler32-zlib) [ 1034.915496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1034.939982] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1042.829178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1043.493074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e network_timeout -r 1 [ 1043.522640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e network_timeout -r 1 [ 1043.551509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1043.565359] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1043.718688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.738685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1043.755546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.775792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.797641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.832128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.865367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.903689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.928286] LNet: 66094:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.948814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.976832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1044.007742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1044.033241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1044.065664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.091175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1044.306640] LNet: 66137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.311549] LNetError: 66137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.349506] LNet: Removed LNI 192.168.204.37@tcp [ 1045.666471] Key type .llcrypt unregistered [ 1045.668221] Key type ._llcrypt unregistered [ 1061.419334] Key type ._llcrypt registered [ 1061.424049] Key type .llcrypt registered [ 1061.646727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.666806] alg: No test for adler32 (adler32-zlib) [ 1062.583911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1062.623735] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1069.140952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1076.767549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.836440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1076.881693] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1078.404272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e network_timeout -r 1 [ 1078.462337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e network_timeout -r 1 [ 1078.516924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e network_timeout -r 1 [ 1078.566554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e network_timeout -r 1 [ 1078.626727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e network_timeout -r 1 [ 1078.669620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e network_timeout -r 1 [ 1078.704552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e network_timeout -r 1 [ 1078.735463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e network_timeout -r 1 [ 1078.774880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1078.805624] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1079.014035] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 1079.029524] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.137@tcp [ 1079.074657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.102752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.131581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.154300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.175577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.205919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.237883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.263628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.277863] LNet: 67021:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.293831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.319580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.346487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.369636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.406223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.473121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.938532] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1083.950627] LNet: Skipped 2 previous similar messages [ 1084.445997] LNet: 67099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.453072] LNetError: 67099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.464895] LNet: Removed LNI 192.168.204.37@tcp [ 1084.479965] LNet: Removed LNI 192.168.204.37@tcp1 [ 1084.898857] Key type .llcrypt unregistered [ 1084.901830] Key type ._llcrypt unregistered [ 1099.294955] Key type ._llcrypt registered [ 1099.297410] Key type .llcrypt registered [ 1100.446644] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:13:11 (1786371191) [ 1100.891531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.903637] alg: No test for adler32 (adler32-zlib) [ 1101.884857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.948859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1101.988301] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1102.756639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1102.780706] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1103.538486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1103.594684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1103.626210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1103.650813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1103.677205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1103.704803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1103.731448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.37@tcp [ 1103.748441] LNetError: 68201:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.37@tcp to recovery [ 1103.967309] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 1103.967405] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1106.015368] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1107.256206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1110.111895] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1110.660742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1110.685765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1110.729828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1110.746912] LNet: 68308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.753766] LNetError: 68308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.757903] LNet: Removed LNI 192.168.204.37@tcp [ 1110.765764] LNet: Removed LNI 192.168.204.37@tcp1 [ 1110.781424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1110.824753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1110.851215] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1111.474262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1111.499805] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1112.218082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1112.251787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.365348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1112.444992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1112.498033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1112.547285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1112.597805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1112.633198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.37@tcp [ 1112.654526] LNetError: 68580:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.37@tcp to recovery [ 1112.864494] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1115.068811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1119.007763] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1119.013839] LNet: Skipped 1 previous similar message [ 1124.281715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1124.336902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.374766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.747805] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:13:42 (1786371222) [ 1131.847519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1131.894827] LNet: 69299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.901446] LNetError: 69299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.908660] LNetError: 69299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1131.919598] LNet: Removed LNI 192.168.204.37@tcp [ 1131.935950] LNet: Removed LNI 192.168.204.37@tcp1 [ 1131.961155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.039137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1132.069409] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1132.670282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1132.685581] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1133.279202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1133.332797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1133.357708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1133.393647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1133.423357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1133.451605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1133.481913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.37@tcp --health 0 [ 1133.570515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1134.048295] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.37@tcp [ 1134.048390] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1134.069446] LNet: Skipped 1 previous similar message [ 1148.900452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1149.061944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.090602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.133744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.37@tcp --health 500 [ 1149.252754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1156.004974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1156.093355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1156.114773] LNet: 69790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.120369] LNetError: 69790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.130353] LNetError: 69790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1156.139321] LNet: Removed LNI 192.168.204.37@tcp [ 1156.148317] LNet: Removed LNI 192.168.204.37@tcp1 [ 1156.169967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1156.221338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1156.242495] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1156.973501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1157.010280] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1157.964260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1157.996779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.053192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1158.084659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1158.112860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1158.152263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1158.197444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1158.232899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.37@tcp --health 0 [ 1159.263649] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1159.267962] LNet: Skipped 3 previous similar messages [ 1159.587848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1169.959509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1169.987465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.011987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.044150] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:14:26 (1786371266) [ 1176.437921] LNet: 70836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.443931] LNetError: 70836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.448390] LNetError: 70836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1176.453750] LNet: Removed LNI 192.168.204.37@tcp [ 1176.461117] LNet: Removed LNI 192.168.204.37@tcp1 [ 1176.975665] Key type .llcrypt unregistered [ 1176.977719] Key type ._llcrypt unregistered [ 1182.098630] Key type ._llcrypt registered [ 1182.101444] Key type .llcrypt registered [ 1182.319435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1182.330883] alg: No test for adler32 (adler32-zlib) [ 1183.206332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1183.225117] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1190.957636] Lustre: DEBUG MARKER: Initial discovery [ 1191.106731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.137@tcp [ 1193.230155] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1193.286383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1193.343303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.137@tcp [ 1193.364966] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1193.383396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.430731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.137@tcp -r 1 -m GET -l 3 [ 1193.503280] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 1193.509103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.137@tcp -r 1 -m GET -e local_error [ 1193.549862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.137@tcp -r 1 -m PUT -l 6 [ 1194.701526] Lustre: DEBUG MARKER: Force oleg437-client.virtnet to discover 192.168.204.137@tcp (in background) [ 1194.763300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.137@tcp [ 1195.925735] Lustre: DEBUG MARKER: Force oleg437-server to discover 192.168.204.37@tcp [ 1197.856975] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1198.247301] Lustre: DEBUG MARKER: Wait for 71595 [ 1199.585687] Lustre: DEBUG MARKER: Finished wait on 71595 [ 1199.664542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1199.713560] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1199.725853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1200.028110] LNet: 72056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.038213] LNetError: 72056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.062811] LNet: Removed LNI 192.168.204.37@tcp [ 1201.386619] Key type .llcrypt unregistered [ 1201.390304] Key type ._llcrypt unregistered [ 1214.984891] Key type ._llcrypt registered [ 1214.987752] Key type .llcrypt registered [ 1216.102247] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:15:06 (1786371306) [ 1216.791917] Key type .llcrypt unregistered [ 1216.794254] Key type ._llcrypt unregistered [ 1221.615655] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1221.759125] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1221.764599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1221.779894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1224.282686] Key type ._llcrypt registered [ 1224.286631] Key type .llcrypt registered [ 1224.421945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1224.432348] alg: No test for adler32 (adler32-zlib) [ 1225.295192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.337602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1225.351814] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1225.831695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1225.844939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1232.900526] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:15:23 (1786371323) [ 1233.105776] LNet: 74138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.111977] LNetError: 74138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1233.120548] LNet: Removed LNI 10.1.2.3@tcp [ 1233.131554] LNet: Removed LNI 192.168.204.37@tcp [ 1233.827371] Key type .llcrypt unregistered [ 1233.829498] Key type ._llcrypt unregistered [ 1240.106349] Key type ._llcrypt registered [ 1240.109858] Key type .llcrypt registered [ 1240.256191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.264819] alg: No test for adler32 (adler32-zlib) [ 1241.140464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.195133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1241.228223] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1241.786452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1241.802598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1242.821982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.37@tcp [ 1243.873150] LNetError: Unexpected error -22 connecting to 192.168.204.37@tcp at host 192.168.204.37:988 [ 1249.163912] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:15:39 (1786371339) [ 1249.284465] LNet: 75267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.294520] LNetError: 75267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1249.310274] LNet: Removed LNI 10.1.2.3@tcp [ 1249.334068] LNet: Removed LNI 192.168.204.37@tcp [ 1249.927406] Key type .llcrypt unregistered [ 1249.929200] Key type ._llcrypt unregistered [ 1256.471611] Key type ._llcrypt registered [ 1256.476572] Key type .llcrypt registered [ 1256.628202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1256.645973] alg: No test for adler32 (adler32-zlib) [ 1257.522596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.588103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1257.605336] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1258.228064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1258.248282] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1259.038825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.37@tcp1 --nid 192.168.204.37@tcp2 [ 1262.265712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp1 192.168.204.37@tcp2 [ 1262.282767] LNetError: 76115:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.37@tcp1 to 192.168.204.37@tcp2 - Route table may be misconfigured [ 1262.303843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp2 192.168.204.37@tcp1 [ 1267.826068] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:15:58 (1786371358) [ 1268.044254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1268.061317] LNet: 76689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.067206] LNetError: 76689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.076054] LNet: Removed LNI 192.168.204.37@tcp1 [ 1269.098358] LNet: Removed LNI 192.168.204.37@tcp2 [ 1269.137809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.175848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.193577] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1269.912812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1269.932331] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1270.604948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1270.635489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.37@tcp [ 1270.663312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp1 192.168.204.37@tcp1 [ 1270.699254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.37@tcp -d 192.168.204.37@tcp -e network_timeout [ 1270.723972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -e network_timeout [ 1270.751714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -e network_timeout [ 1270.779828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -e network_timeout [ 1270.807896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1270.826633] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1270.841873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1270.916372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1271.014296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1271.202722] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 1276.821687] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:16:07 (1786371367) [ 1276.901501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1276.919604] LNet: 77543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1276.928049] LNetError: 77543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.937487] LNetError: 77543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1276.942414] LNet: Removed LNI 192.168.204.37@tcp [ 1277.991306] LNet: Removed LNI 192.168.204.37@tcp1 [ 1278.031174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.106220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1278.417652] LNet: 77592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.843457] Key type .llcrypt unregistered [ 1278.846307] Key type ._llcrypt unregistered [ 1289.008904] Key type ._llcrypt registered [ 1289.013969] Key type .llcrypt registered [ 1290.071942] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:16:20 (1786371380) [ 1290.474647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.491438] alg: No test for adler32 (adler32-zlib) [ 1291.355168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.560233] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1293.962703] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:16:24 (1786371384) [ 1294.187336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1294.206360] LNet: 78689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.224177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.276746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1294.297656] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1294.882390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.901274] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1295.824323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1295.860271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp1 [ 1295.890094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp1 [ 1301.737220] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:16:32 (1786371392) [ 1303.445162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg437-server [ 1304.895090] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:16:35 (1786371395) [ 1306.954347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg437-server [ 1308.755817] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:16:39 (1786371399) [ 1310.263951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg437-server [ 1311.609590] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:16:42 (1786371402) [ 1313.317927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg437-server [ 1315.112326] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:16:45 (1786371405) [ 1316.839781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg437-server [ 1318.244510] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:16:49 (1786371409) [ 1320.029732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg437-server [ 1321.573983] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:16:52 (1786371412) [ 1323.080885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg437-server [ 1324.949987] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:16:55 (1786371415) [ 1326.398176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg437-server [ 1327.739607] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:16:58 (1786371418) [ 1329.074883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg437-server [ 1330.661719] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:17:01 (1786371421) [ 1330.844871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.900708] LNet: 82560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.912055] LNetError: 82560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.917635] LNet: Removed LNI 192.168.204.37@tcp [ 1331.942995] LNet: Removed LNI 192.168.204.37@tcp1 [ 1331.973795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.057299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1332.078265] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1332.734538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1332.810082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1332.962754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.978891] LNet: 82712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.984954] LNetError: 82712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.990680] LNetError: 82712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1334.052668] LNet: Removed LNI 192.168.204.37@tcp [ 1334.070334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1334.134286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1334.158052] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1334.824757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1334.999344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.021662] LNet: 82858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.027391] LNetError: 82858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.041181] LNet: Removed LNI 192.168.204.37@tcp [ 1335.057448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.096815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1335.119794] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1335.880438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1335.952641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1336.091764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1337.126591] LNet: Removed LNI 192.168.204.37@tcp [ 1337.164731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.255564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1337.306254] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1337.938353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1338.172902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1338.199258] LNetError: 83150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.202399] LNetError: 83150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1338.216719] LNet: Removed LNI 192.168.204.37@tcp [ 1338.244666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.302686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1338.321850] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1338.897899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1338.961860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1339.201891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.230326] LNet: 83298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.236902] LNet: 83298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1340.262275] LNet: Removed LNI 192.168.204.37@tcp [ 1340.302293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.353536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1340.387496] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1341.038571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1341.297940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1342.373686] LNet: Removed LNI 192.168.204.37@tcp [ 1342.393310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.469244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1342.489294] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1343.069796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1343.152649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1343.361255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1343.378980] LNetError: 83590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.388085] LNetError: 83590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1344.422146] LNet: Removed LNI 192.168.204.37@tcp [ 1344.456681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.551784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1344.589786] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1345.563233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1345.803268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.854710] LNet: Removed LNI 192.168.204.37@tcp [ 1346.879443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.979509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1347.010831] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1347.571890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1347.662465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1347.898995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1347.917512] LNet: 83883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.925781] LNet: 83883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1348.964761] LNet: Removed LNI 192.168.204.37@tcp [ 1348.991395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.050519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1349.072219] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1349.739649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1349.959952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1351.013825] LNet: Removed LNI 192.168.204.37@tcp [ 1351.042861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.133413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1351.153687] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1351.753737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1351.805751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1352.083132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1352.097210] LNetError: 84176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.101640] LNetError: 84176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1353.126359] LNet: Removed LNI 192.168.204.37@tcp [ 1353.153206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.202463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1353.234776] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1353.828133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1354.051194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.109892] LNet: Removed LNI 192.168.204.37@tcp [ 1355.146672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1355.262318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1355.302478] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1356.085860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1356.132778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.161776] LNet: Removed LNI 192.168.204.37@tcp [ 1356.180469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.242499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.268899] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1356.941208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1357.004372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.028268] LNet: Removed LNI 192.168.204.37@tcp [ 1357.055906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.098334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1357.176617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.219272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.265847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1357.334927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.372550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.409599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1357.495411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.535710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.587573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1357.605534] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1358.335995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.358469] LNet: Removed LNI 192.168.204.37@tcp [ 1358.390892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.457421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1363.932839] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:17:34 (1786371454) [ 1364.005063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.026859] LNet: 85335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.031816] LNet: 85335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1364.065845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.137672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1364.164686] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/180] [ 1365.110450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.133929] LNet: Removed LNI 192.168.204.37@tcp231 [ 1365.154255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.204305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1365.231381] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/180] [ 1365.237671] LNet: 85489:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1365.535151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1365.564318] LNet: Removed LNI 192.168.204.37@tcp231 [ 1365.587346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1365.619124] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/180] [ 1366.432891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.451161] LNet: Removed LNI 192.168.204.37@tcp231 [ 1366.465321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.522238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1366.545420] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/0] [ 1366.551260] LNet: 85641:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1366.851319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1366.875606] LNet: Removed LNI 192.168.204.37@tcp231 [ 1366.892252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1366.923327] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/0] [ 1367.707431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.741636] LNet: Removed LNI 192.168.204.37@tcp231 [ 1367.758526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.822367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1367.849191] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/60] [ 1367.856586] LNet: 85792:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1368.225048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1368.254531] LNet: Removed LNI 192.168.204.37@tcp231 [ 1368.269600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1368.299776] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/60] [ 1369.083561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1369.100154] LNetError: 85935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.104660] LNetError: 85935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1369.116742] LNet: Removed LNI 192.168.204.37@tcp231 [ 1369.128495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.175969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1369.193709] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/180] [ 1369.613308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1369.638652] LNet: Removed LNI 192.168.204.37@tcp231 [ 1369.662302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1369.694394] LNet: Added LNI 192.168.204.37@tcp231 [8/256/0/180] [ 1375.428241] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:17:46 (1786371466) [ 1375.605235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1375.647559] LNet: Removed LNI 192.168.204.37@tcp231 [ 1375.666264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1375.723765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1375.749101] LNet: Added LNI 192.168.204.37@tcp232 [8/256/0/180] [ 1376.250874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1376.278653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1376.302476] LNet: Removed LNI 192.168.204.37@tcp232 [ 1376.321064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1376.348684] LNet: Added LNI 192.168.204.37@tcp232 [8/256/0/180] [ 1376.901929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1376.938322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1376.978420] LNet: Removed LNI 192.168.204.37@tcp232 [ 1376.992894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1377.012608] LNet: Added LNI 192.168.204.37@tcp232 [8/256/0/180] [ 1377.677573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1377.710179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1377.731268] LNet: Removed LNI 192.168.204.37@tcp232 [ 1377.744249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1377.766611] LNet: Added LNI 192.168.204.37@tcp232 [8/256/0/180] [ 1378.357438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1378.413538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1378.457818] LNet: Removed LNI 192.168.204.37@tcp232 [ 1378.476673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1383.341672] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:17:54 (1786371474) [ 1384.392457] Key type .llcrypt unregistered [ 1384.394351] Key type ._llcrypt unregistered [ 1388.799806] Key type ._llcrypt registered [ 1388.801683] Key type .llcrypt registered [ 1388.930071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1388.941356] alg: No test for adler32 (adler32-zlib) [ 1389.793480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1389.814334] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1395.283407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1402.544579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.606513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1402.633848] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1403.944463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_interrupt -r 2 [ 1403.984487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_interrupt -r 2 [ 1404.021874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_interrupt -r 2 [ 1404.082651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_interrupt -r 2 [ 1404.110880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_interrupt -r 2 [ 1404.146763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_interrupt -r 2 [ 1404.196798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_interrupt -r 2 [ 1404.229644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_interrupt -r 2 [ 1404.260446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.137@tcp [ 1404.278483] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1404.283695] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 1404.562472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1404.620652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1404.660667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1404.698923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1404.755598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1404.790480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1404.816825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1404.861045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1404.892666] LNet: 88447:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1404.912645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1404.945693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1404.977969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.002672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.028806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.061321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.524144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_dropped -r 2 [ 1405.555074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_dropped -r 2 [ 1405.596646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_dropped -r 2 [ 1405.635889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_dropped -r 2 [ 1405.691750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_dropped -r 2 [ 1405.741563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_dropped -r 2 [ 1405.778860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_dropped -r 2 [ 1405.809532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_dropped -r 2 [ 1405.847870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.137@tcp [ 1405.877881] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1406.124578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1406.152196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1406.173471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1406.213793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1406.261723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1406.291607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1406.327992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1406.368597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1406.413953] LNet: 88567:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1406.441512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1406.482348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1406.516089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1406.539659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1406.571333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1406.596378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1407.011945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_aborted -r 2 [ 1407.051565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_aborted -r 2 [ 1407.090814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_aborted -r 2 [ 1407.119769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_aborted -r 2 [ 1407.157163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_aborted -r 2 [ 1407.203675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_aborted -r 2 [ 1407.241508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_aborted -r 2 [ 1407.281657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_aborted -r 2 [ 1407.327869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.137@tcp [ 1407.376708] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1407.600737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1407.624072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1407.647240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1407.671536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1407.698604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1407.720950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1407.744538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1407.767170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1407.788216] LNet: 88687:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1407.819813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1407.856269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1407.898795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1407.926783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1407.958148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1407.988682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1408.515864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_no_route -r 2 [ 1408.542425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_no_route -r 2 [ 1408.571918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_no_route -r 2 [ 1408.606831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_no_route -r 2 [ 1408.641899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_no_route -r 2 [ 1408.689777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_no_route -r 2 [ 1408.735083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_no_route -r 2 [ 1408.767414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_no_route -r 2 [ 1408.806428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.137@tcp [ 1409.046772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1409.069231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1409.092900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1409.120468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1409.150851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1409.177484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1409.197341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1409.225934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1409.250186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1409.288848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1409.313826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.339995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.380267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1409.418151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1409.880697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e local_timeout -r 2 [ 1409.915870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e local_timeout -r 2 [ 1409.949559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e local_timeout -r 2 [ 1409.987828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e local_timeout -r 2 [ 1410.026130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e local_timeout -r 2 [ 1410.075895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e local_timeout -r 2 [ 1410.113458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e local_timeout -r 2 [ 1410.156658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e local_timeout -r 2 [ 1410.216814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.137@tcp [ 1410.242707] LNet: There was an unexpected network error while writing to 192.168.204.137: rc = -22 [ 1410.250289] LNet: Skipped 1 previous similar message [ 1410.415853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1410.446602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1410.473332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1410.508385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1410.536884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.559587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1410.586992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.609735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1410.624745] LNet: 88928:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1410.629354] LNet: 88928:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1410.642818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1410.666740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1410.687424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1410.709209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.742468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1410.771864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1411.174214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp -m GET -e remote_dropped -r 2 [ 1411.208499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.137@tcp1 -m GET -e remote_dropped -r 2 [ 1411.243334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp -m GET -e remote_dropped -r 2 [ 1411.272974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp -d 192.168.204.37@tcp1 -m GET -e remote_dropped -r 2 [ 1411.307975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp -m GET -e remote_dropped -r 2 [ 1411.335801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.37@tcp1 -d 192.168.204.137@tcp1 -m GET -e remote_dropped -r 2 [ 1411.366947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp -m GET -e remote_dropped -r 2 [ 1411.393384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.37@tcp1 -d 192.168.204.37@tcp1 -m GET -e remote_dropped -r 2 [ 1411.420914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.37@tcp 192.168.204.137@tcp [ 1411.439426] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.137@tcp [ 1411.631831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1411.649720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1411.669462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1411.699971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1411.729713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1411.764539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1411.793760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1411.817708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1411.845176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1411.866345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1411.886813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1411.914347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1411.945296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.969664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.788478] LNet: 89126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.794426] LNetError: 89126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.863529] LNet: Removed LNI 192.168.204.37@tcp [ 1416.865690] LNetError: 89126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.879565] LNet: Removed LNI 192.168.204.37@tcp1 [ 1417.344851] Key type .llcrypt unregistered [ 1417.346916] Key type ._llcrypt unregistered [ 1429.873896] Key type ._llcrypt registered [ 1429.875877] Key type .llcrypt registered [ 1430.963866] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:18:41 (1786371521) [ 1431.264053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.276638] alg: No test for adler32 (adler32-zlib) [ 1432.156121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.197924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.220616] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1432.929535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1432.951479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.37@tcp [ 1432.968349] LNetError: 90105:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.37@tcp to recovery [ 1433.051571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.37@tcp [ 1433.150910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1438.114493] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:18:49 (1786371529) [ 1438.290570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.314414] LNet: 90696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.323386] LNetError: 90696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.337652] LNet: Removed LNI 192.168.204.37@tcp [ 1438.353142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.391853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.413538] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1438.961033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1438.983292] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1439.740949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.37@tcp [ 1439.757565] LNetError: 90962:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.37@tcp to recovery [ 1440.415355] LNet: 1 local NIs in recovery (showing 1): 192.168.204.37@tcp [ 1440.420492] LNet: There was an unexpected network error while writing to 192.168.204.37: rc = -22 [ 1440.913947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.37@tcp [ 1447.921458] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:18:59 (1786371539) [ 1449.396562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg437-server [ 1450.703328] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:19:01 (1786371541) [ 1452.273442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg437-server [ 1453.645651] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:19:04 (1786371544) [ 1455.216175] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg437-server [ 1457.054378] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:19:07 (1786371547) [ 1457.236208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.273043] LNet: 92591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.281599] LNetError: 92591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.291844] LNet: Removed LNI 192.168.204.37@tcp [ 1457.307417] LNet: Removed LNI 192.168.204.37@tcp1 [ 1457.337947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.411151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.445205] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1458.122523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1458.175647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1458.216334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1458.281596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1458.324954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1458.357597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1463.681215] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:19:14 (1786371554) [ 1463.868174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.889352] LNet: 93316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.895792] LNetError: 93316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.901844] LNetError: 93316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1463.926658] LNet: Removed LNI 192.168.204.37@tcp [ 1463.954253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.023295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.038940] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1469.356990] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:19:20 (1786371560) [ 1469.481943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.502742] LNet: 94018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.511201] LNetError: 94018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.536659] LNet: Removed LNI 192.168.204.37@tcp [ 1469.554636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.602073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.631110] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1475.227675] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:19:26 (1786371566) [ 1476.799552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1478.425890] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:19:29 (1786371569) [ 1478.829206] LNet: 95053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.834946] LNetError: 95053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.854960] LNet: Removed LNI 192.168.204.37@tcp [ 1479.425867] Key type .llcrypt unregistered [ 1479.431374] Key type ._llcrypt unregistered [ 1484.598281] Key type ._llcrypt registered [ 1484.612811] Key type .llcrypt registered [ 1484.829608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1484.842972] alg: No test for adler32 (adler32-zlib) [ 1485.818719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1485.860494] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1491.794674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1495.452582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.137@tcp [ 1495.850748] LNet: 95478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.866384] LNetError: 95478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.892736] LNet: Removed LNI 192.168.204.37@tcp [ 1496.338419] Key type .llcrypt unregistered [ 1496.340689] Key type ._llcrypt unregistered [ 1505.528666] Key type ._llcrypt registered [ 1505.531473] Key type .llcrypt registered [ 1506.638447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1508.123610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1509.476892] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:20:00 (1786371600) [ 1510.976384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1512.602803] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:20:03 (1786371603) [ 1514.180821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg437-server [ 1515.556939] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:20:06 (1786371606) [ 1516.793294] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg437-server [ 1518.270209] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:20:09 (1786371609) [ 1519.458587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.482110] alg: No test for adler32 (adler32-zlib) [ 1520.551145] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 1520.739991] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1522.391234] Key type lgssc registered [ 1523.337935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.370103] Key type lgssc unregistered [ 1537.565833] LNet: 98655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.571807] LNetError: 98655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.585868] LNet: Removed LNI 192.168.204.37@tcp [ 1538.119888] Key type .llcrypt unregistered [ 1538.121837] Key type ._llcrypt unregistered [ 1549.790345] Key type ._llcrypt registered [ 1549.791450] Key type .llcrypt registered [ 1550.954429] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:20:41 (1786371641) [ 1552.068969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.100097] alg: No test for adler32 (adler32-zlib) [ 1553.162327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.371700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1553.534161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1553.583232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1553.636440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1564.760926] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:20:54 (1786371654) [ 1565.324509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.398521] LNet: 100072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.462367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.586494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1565.629507] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1567.991391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.029308] LNet: 100216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.044978] LNetError: 100216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.065682] LNet: Removed LNI 192.168.204.37@tcp [ 1568.124498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.244952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1568.293439] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1569.229777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.266362] LNet: 100244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.277685] LNetError: 100244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.292601] LNet: Removed LNI 192.168.204.37@tcp [ 1569.328650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.447602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1569.509975] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1579.505035] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:21:09 (1786371669) [ 1579.862131] LNet: 100835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.894843] LNetError: 100835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.921459] LNet: Removed LNI 192.168.204.37@tcp [ 1580.762066] Key type .llcrypt unregistered [ 1580.769587] Key type ._llcrypt unregistered [ 1591.199572] Key type ._llcrypt registered [ 1591.210438] Key type .llcrypt registered [ 1591.634485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1591.669643] alg: No test for adler32 (adler32-zlib) [ 1592.660383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.778455] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1592.781274] LNet: *** cfs_fail_loc=e003, val=100*** [ 1594.227873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1594.279844] LNet: *** cfs_fail_loc=e003, val=100*** [ 1594.290353] LNet: Skipped 2 previous similar messages [ 1594.300685] Ping reply received with mapping: [ 1594.306355] found metadata reply NI[192.168.204.37@tcp] [ 1594.310873] 25 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1594.316767] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1594.321926] 5a 5a 5a 5a 5a 5a 5a 5a [ 1594.529582] LNet: 101237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.546883] LNetError: 101237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.621892] LNet: Removed LNI 192.168.204.37@tcp [ 1597.011525] Key type .llcrypt unregistered [ 1597.015217] Key type ._llcrypt unregistered [ 1615.098263] Key type ._llcrypt registered [ 1615.100796] Key type .llcrypt registered [ 1616.983137] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:21:46 (1786371706) [ 1617.814197] Key type .llcrypt unregistered [ 1617.815809] Key type ._llcrypt unregistered [ 1628.869662] Key type ._llcrypt registered [ 1628.883102] Key type .llcrypt registered [ 1629.333567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1629.368216] alg: No test for adler32 (adler32-zlib) [ 1630.855683] LNetError: 102297:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1631.396603] Key type .llcrypt unregistered [ 1631.401491] Key type ._llcrypt unregistered [ 1637.852796] Key type ._llcrypt registered [ 1637.854969] Key type .llcrypt registered [ 1639.366431] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:22:09 (1786371729) [ 1640.023481] Key type .llcrypt unregistered [ 1640.030595] Key type ._llcrypt unregistered [ 1648.902070] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1648.904296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1651.527112] Key type ._llcrypt registered [ 1651.532943] Key type .llcrypt registered [ 1651.920845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1651.948331] alg: No test for adler32 (adler32-zlib) [ 1652.878277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.962873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1662.721546] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:22:33 (1786371753) [ 1662.870243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.900160] LNet: 104080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.929492] LNetError: 104080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1662.952464] LNet: Removed LNI 10.1.2.3@tcp [ 1663.014977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.112890] Key type .llcrypt unregistered [ 1664.115092] Key type ._llcrypt unregistered [ 1671.165816] Key type ._llcrypt registered [ 1671.168914] Key type .llcrypt registered [ 1671.335742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1671.348668] alg: No test for adler32 (adler32-zlib) [ 1672.409343] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1672.547397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1672.683489] LNet: 104378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.708319] LNetError: 104378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.722808] LNet: Removed LNI 192.168.204.37@tcp [ 1673.251665] Key type .llcrypt unregistered [ 1673.253216] Key type ._llcrypt unregistered [ 1684.819147] Key type ._llcrypt registered [ 1684.822184] Key type .llcrypt registered [ 1685.887389] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:22:56 (1786371776) [ 1686.801602] Key type .llcrypt unregistered [ 1686.804277] Key type ._llcrypt unregistered [ 1693.686718] Key type ._llcrypt registered [ 1693.697740] Key type .llcrypt registered [ 1693.893914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1693.905703] alg: No test for adler32 (adler32-zlib) [ 1701.751411] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:23:12 (1786371792) [ 1702.205203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.268564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.398606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.434223] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1703.411413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.461323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1703.504682] LNet: Removed LNI 192.168.204.37@tcp [ 1703.787898] LNet: 106134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.109467] Key type .llcrypt unregistered [ 1704.111815] Key type ._llcrypt unregistered [ 1718.027607] Key type ._llcrypt registered [ 1718.030869] Key type .llcrypt registered [ 1719.430105] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:23:29 (1786371809) [ 1720.228853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1720.249752] alg: No test for adler32 (adler32-zlib) [ 1721.117983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.174885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.194908] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1721.806388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1721.836992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.37@tcp [ 1721.854090] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1721.858302] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1721.861357] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1721, 0, 0 [ 1721.866075] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1721.868764] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1721.872050] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1721.875165] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1721, 0, 0 [ 1721.878897] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1721.881915] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1721.886628] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1721.888854] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1721, 0, 1771 [ 1721.892501] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1721.895042] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1721.901703] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1721.904907] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1721, 0, 1771 [ 1721.909500] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1728.959631] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:23:39 (1786371819) [ 1729.580185] LNet: 107673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.586758] LNetError: 107673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.663047] LNet: Removed LNI 192.168.204.37@tcp [ 1731.462457] Key type .llcrypt unregistered [ 1731.465750] Key type ._llcrypt unregistered [ 1739.606839] Key type ._llcrypt registered [ 1739.615038] Key type .llcrypt registered [ 1739.969569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1739.992934] alg: No test for adler32 (adler32-zlib) [ 1740.939094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1740.959696] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1749.485701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 1758.196764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.277620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1758.341336] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 1759.659466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1759.903722] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1759.926131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1759.940874] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1762.572814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1762.614954] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1775.383850] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:24:25 (1786371865) [ 1775.590301] LNet: 109177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.605166] LNetError: 109177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.618399] LNet: Removed LNI 192.168.204.37@tcp [ 1776.671305] LNet: Removed LNI 192.168.204.37@tcp1 [ 1776.675725] LNetError: 109177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1776.685511] LNetError: 109177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1776.705729] LNet: Removed LNI 10.1.2.3@tcp99 [ 1777.363723] Key type .llcrypt unregistered [ 1777.366190] Key type ._llcrypt unregistered [ 1784.562666] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1784.575393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1787.675990] Key type ._llcrypt registered [ 1787.682942] Key type .llcrypt registered [ 1788.019025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1788.042482] alg: No test for adler32 (adler32-zlib) [ 1789.225679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.312472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1789.342582] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1790.128917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1790.173936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1792.379422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1792.412796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1792.448721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1792.462970] LNetError: 109935:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1792.496811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1792.568982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1792.958750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1793.013753] LNet: 109955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.017347] LNetError: 109955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1793.020953] LNet: Removed LNI 10.1.2.3@tcp [ 1794.087983] LNet: Removed LNI 192.168.204.37@tcp [ 1794.836785] Key type .llcrypt unregistered [ 1794.839550] Key type ._llcrypt unregistered [ 1809.063738] Key type ._llcrypt registered [ 1809.066016] Key type .llcrypt registered [ 1810.548919] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:25:00 (1786371900) [ 1811.312990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.344570] alg: No test for adler32 (adler32-zlib) [ 1812.286945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.359344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.410386] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1813.106312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1813.184581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg437-client@tcp [ 1819.516660] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:25:10 (1786371910) [ 1819.759179] LNet: 111493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.769873] LNetError: 111493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.838240] LNet: Removed LNI 192.168.204.37@tcp [ 1821.460670] Key type .llcrypt unregistered [ 1821.465782] Key type ._llcrypt unregistered [ 1829.393609] Key type ._llcrypt registered [ 1829.401452] Key type .llcrypt registered [ 1829.665207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.682604] alg: No test for adler32 (adler32-zlib) [ 1830.551377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1830.576211] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1830.611548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1837.200268] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:25:27 (1786371927) [ 1839.072139] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1840.966240] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:25:31 (1786371931) [ 1842.543888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1844.473901] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:25:35 (1786371935) [ 1844.820706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.841925] LNet: 112917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.850518] LNetError: 112917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.866574] LNet: Removed LNI 192.168.204.37@tcp [ 1844.895117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.959809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.977916] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1845.606699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1845.647616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1851.470682] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:25:42 (1786371942) [ 1851.662518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.719739] LNet: 113628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.728863] LNetError: 113628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.746232] LNet: Removed LNI 192.168.204.37@tcp [ 1851.766687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.811883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1851.839638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1852.890580] Key type .llcrypt unregistered [ 1852.899508] Key type ._llcrypt unregistered [ 1868.168883] Key type ._llcrypt registered [ 1868.171017] Key type .llcrypt registered [ 1869.395824] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:25:59 (1786371959) [ 1870.063711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.097584] alg: No test for adler32 (adler32-zlib) [ 1870.979278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.027755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.044989] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1871.820981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1871.891438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1880.619877] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:26:10 (1786371970) [ 1880.737862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.783530] LNet: 115183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.794735] LNetError: 115183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.807041] LNet: Removed LNI 192.168.204.37@tcp [ 1880.840114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.923360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1880.957513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1889.295096] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:26:19 (1786371979) [ 1889.817688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.852912] LNet: 115770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.889907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.008471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1890.455110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1890.497585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1890.545114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1891.010893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1891.049721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1891.138479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.158836] LNet: 115804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.185565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.263241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1891.597264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1891.632809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1891.693339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1892.051615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1892.082905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1900.270229] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:26:30 (1786371990) [ 1900.563441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.652572] LNet: 116402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.758934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.851692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.901996] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1902.157697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1902.193393] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1903.531277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1903.619700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1903.808706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.827450] LNetError: 116684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.832259] LNet: Removed LNI 192.168.204.37@tcp [ 1903.841791] LNet: Removed LNI 192.168.204.37@tcp2 [ 1903.874671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.958273] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1903.964404] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1903.969279] LNet: 116690:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1904.552260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1904.895804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.928794] LNet: 116727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.933603] LNet: 116727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1904.944861] LNetError: 116727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.955355] LNetError: 116727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1904.976724] LNet: Removed LNI 192.168.204.37@tcp [ 1904.989390] LNet: Removed LNI 192.168.204.37@tcp2 [ 1905.026843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.130232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.181253] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1906.373133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1906.442513] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1908.032666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1908.097642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1908.166591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1908.464401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.500235] LNetError: 117012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.509815] LNetError: 117012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1909.535190] LNet: Removed LNI 192.168.204.37@tcp [ 1910.565968] LNet: Removed LNI 192.168.204.37@tcp2 [ 1910.618705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.710883] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1910.718041] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1910.723789] LNet: 117018:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1911.287708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1911.523330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.542952] LNet: Removed LNI 192.168.204.37@tcp [ 1912.614076] LNet: Removed LNI 192.168.204.37@tcp2 [ 1912.642090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.726757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.770050] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1913.694951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1913.713320] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1914.603333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1914.694901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1914.906918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.926276] LNet: 117335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.930525] LNet: 117335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1914.937596] LNetError: 117335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.945294] LNetError: 117335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1914.952609] LNet: Removed LNI 192.168.204.37@tcp [ 1914.964824] LNet: Removed LNI 192.168.204.37@tcp2 [ 1914.996944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.075989] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1915.085627] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1915.090794] LNet: 117341:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1915.823784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1916.117645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.215439] LNet: Removed LNI 192.168.204.37@tcp [ 1917.228854] LNet: Removed LNI 192.168.204.37@tcp2 [ 1917.253376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.352083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.390201] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1918.571946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1918.587552] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1919.779296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1919.844836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1919.987685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1920.379393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.439285] LNet: Removed LNI 192.168.204.37@tcp [ 1922.469934] LNet: Removed LNI 192.168.204.37@tcp2 [ 1922.514323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.653955] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1922.668402] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1922.679435] LNet: 117669:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1923.776174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1924.274198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1924.323907] LNetError: 117706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.335738] LNetError: 117706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1924.350827] LNet: Removed LNI 192.168.204.37@tcp [ 1925.431220] LNet: Removed LNI 192.168.204.37@tcp2 [ 1925.496799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.642607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.722197] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1926.856360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1926.906965] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1928.476245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1928.548981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1928.657655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1929.478795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.527270] LNet: Removed LNI 192.168.204.37@tcp [ 1930.535861] LNet: Removed LNI 192.168.204.37@tcp2 [ 1930.593368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.768421] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1930.792156] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1930.806957] LNet: 118009:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1931.094734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1931.543275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.568931] LNet: 118061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.573876] LNet: 118061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1932.639181] LNet: Removed LNI 192.168.204.37@tcp [ 1932.647917] LNet: Removed LNI 192.168.204.37@tcp2 [ 1932.691326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.779245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.810736] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1933.580218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1933.601571] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1934.654626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1934.686661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1934.714554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1934.762105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1935.300736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.351373] LNet: Removed LNI 192.168.204.37@tcp [ 1937.393444] LNet: Removed LNI 192.168.204.37@tcp2 [ 1937.458924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.571795] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1937.590356] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 1938.235479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1949.354512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1952.362906] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:27:21 (1786372041) [ 1952.556495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.616548] LNetError: 119129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.625442] LNetError: 119129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1952.636344] LNet: Removed LNI 192.168.204.37@tcp [ 1953.703229] LNet: Removed LNI 192.168.204.37@tcp2 [ 1953.726982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.082491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1954.132059] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1954.902923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1954.947898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1955.975843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.996285] LNet: Removed LNI 10.1.2.3@tcp [ 1956.009898] LNet: Removed LNI 192.168.204.37@tcp [ 1956.032172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.112980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1956.141242] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1956.147163] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1956.468706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1956.488630] LNet: Removed LNI 192.168.204.37@tcp [ 1956.499762] LNet: Removed LNI 10.1.2.3@tcp [ 1963.999649] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:27:34 (1786372054) [ 1964.298870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.327734] LNet: 120009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.333542] LNet: 120009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1964.392243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.449483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1964.490945] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1965.462220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1965.502589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.532378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.581181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.625921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.671551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.725884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.768145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.810607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.850687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1965.889640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.37@tcp [ 1966.003221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1973.411587] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:27:43 (1786372063) [ 1973.529937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.629793] LNet: Removed LNI 192.168.204.37@tcp [ 1974.672991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.807925] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1974.991065] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1974.999186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1975.024409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1977.529441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.571371] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 1978.600712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1978.624032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1981.657076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 1981.685744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1981.737300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.37@tcp -m PUT -e local_timeout -r 1 [ 1981.831564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1981.921534] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1981.927919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1982.188212] LNetError: 120809:0:(lib-move.c:4801:lnet_parse()) 192.168.204.37@tcp, src 192.168.204.37@tcp: Dropping PUT (error -108 looking up sender) [ 1982.945990] LNet: Removed LNI 10.1.2.3@tcp [ 1983.977995] LNet: Removed LNI 192.168.204.37@tcp [ 1984.550176] Key type .llcrypt unregistered [ 1984.552145] Key type ._llcrypt unregistered [ 2001.857949] Key type ._llcrypt registered [ 2001.860333] Key type .llcrypt registered [ 2003.595270] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:28:13 (1786372093) [ 2004.228174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.241870] alg: No test for adler32 (adler32-zlib) [ 2005.280421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.935413] LNet: 122125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.822391] Key type .llcrypt unregistered [ 2006.828224] Key type ._llcrypt unregistered [ 2015.925854] Key type ._llcrypt registered [ 2015.927777] Key type .llcrypt registered [ 2016.150419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2016.166321] alg: No test for adler32 (adler32-zlib) [ 2017.152587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2017.179385] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 2027.544817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 2027.738688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2028.849338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.137@tcp [ 2029.341953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2029.432852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2029.471259] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.137@tcp [ 2029.538386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2029.595871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2029.660497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2029.689647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2029.742524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2029.781454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2029.803733] LNet: 122580:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2029.821401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2029.849388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2029.878794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2029.908652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2029.935504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2029.970370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2030.643860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2030.964614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2031.002727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2031.035651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2031.069813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2031.104722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2031.132772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2031.165753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2031.197514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2031.215536] LNet: 122697:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2031.231944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2031.263646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2031.309214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2031.339277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2031.378688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2031.424317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2032.236115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.137@tcp [ 2032.573758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2032.621566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2032.661834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2032.688647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2032.719113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2032.751049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2032.785601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2032.810535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2032.831400] LNet: 122814:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2032.851133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2032.884142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2032.930834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2032.984448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2033.039763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2033.086586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2034.191879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2034.505790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2034.559555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2034.594939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2034.619984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2034.665904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2034.709254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2034.755255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2034.786951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2034.826935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2034.880620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2034.925662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2034.957226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2034.993160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2035.028255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2035.374904] LNet: 122982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.384972] LNetError: 122982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.404614] LNet: Removed LNI 192.168.204.37@tcp [ 2035.866044] Key type .llcrypt unregistered [ 2035.873404] Key type ._llcrypt unregistered [ 2058.485155] Key type ._llcrypt registered [ 2058.489883] Key type .llcrypt registered [ 2059.709947] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:29:10 (1786372150) [ 2060.806389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.826251] alg: No test for adler32 (adler32-zlib) [ 2061.927266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.864341] LNet: 123870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.343115] Key type .llcrypt unregistered [ 2064.348536] Key type ._llcrypt unregistered [ 2072.614946] Key type ._llcrypt registered [ 2072.618342] Key type .llcrypt registered [ 2073.028098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2073.069539] alg: No test for adler32 (adler32-zlib) [ 2074.181134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2074.209923] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 2082.796265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.137@tcp [ 2092.664472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.726715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2092.770867] LNet: Added LNI 192.168.204.37@tcp1 [8/256/0/180] [ 2093.721659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2094.638977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.137@tcp [ 2094.688123] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.137@tcp [ 2094.944680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2094.981118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2095.022609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2095.059849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2095.089291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2095.143284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2095.174688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2095.212321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2095.230275] LNet: 124527:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2095.258655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2095.314799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2095.354866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2095.387278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2095.443798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2095.479739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2096.405950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2096.708904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2096.763321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2096.799186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2096.833788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2096.871761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2096.909040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2096.974083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2097.043184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2097.071812] LNet: 124644:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2097.106527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2097.140592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2097.173830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2097.218771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2097.255688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2097.290923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2098.148477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.137@tcp [ 2098.470331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2098.535610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2098.634635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2098.701110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2098.757396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2098.807195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2098.885478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2098.919085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2098.940727] LNet: 124761:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2098.960405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2099.000364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2099.029755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2099.052772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2099.076179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2099.098574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2100.013889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2100.351991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2100.382543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2100.409275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2100.433116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2100.474945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2100.506809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2100.539639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2100.566300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2100.591742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2100.617409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2100.644578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2100.676464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2100.717646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2100.772539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2105.870637] LNet: 124963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.875219] LNetError: 124963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.881735] LNet: Removed LNI 192.168.204.37@tcp [ 2105.893863] LNet: Removed LNI 192.168.204.37@tcp1 [ 2106.341519] Key type .llcrypt unregistered [ 2106.343145] Key type ._llcrypt unregistered [ 2125.903326] Key type ._llcrypt registered [ 2125.905711] Key type .llcrypt registered [ 2127.170790] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:30:17 (1786372217) [ 2127.709880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2127.727238] alg: No test for adler32 (adler32-zlib) [ 2128.630805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.675628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2128.699953] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 2129.363390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.385858] LNet: Added LNI 192.168.204.37@tcp2 [8/256/0/180] [ 2130.618888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.37@tcp [ 2130.686986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2131.743229] LNet: Removed LNI 192.168.204.37@tcp [ 2138.467947] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:30:28 (1786372228) [ 2140.180283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2142.275600] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:30:32 (1786372232) [ 2150.191540] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:30:40 (1786372240) [ 2158.320269] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:30:48 (1786372248) [ 2165.683549] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:30:56 (1786372256) [ 2173.466475] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:31:04 (1786372264) [ 2180.891347] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:31:11 (1786372271) [ 2188.891343] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:31:19 (1786372279) [ 2195.919213] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:31:26 (1786372286) [ 2203.209105] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:31:33 (1786372293) [ 2210.043580] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:31:40 (1786372300) [ 2218.296143] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:31:48 (1786372308) [ 2224.937253] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:31:55 (1786372315) [ 2231.733921] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:32:02 (1786372322) [ 2238.406231] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:32:09 (1786372329) [ 2245.572314] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:32:16 (1786372336) [ 2255.205796] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:32:25 (1786372345) [ 2262.519136] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:32:33 (1786372353) [ 2270.685089] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:32:41 (1786372361) [ 2270.870589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.902740] LNet: 136656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.907901] LNetError: 136656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.937817] LNet: Removed LNI 192.168.204.37@tcp2 [ 2270.983396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.122783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.155109] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 2272.310861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.329840] LNet: 136801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.335197] LNetError: 136801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.351893] LNet: Removed LNI 192.168.204.37@tcp [ 2272.374633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.432862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2272.459771] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 2279.221996] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:32:49 (1786372369) [ 2287.710312] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:32:58 (1786372378) [ 2294.340493] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:33:05 (1786372385) [ 2301.715996] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:33:12 (1786372392) [ 2308.457893] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:33:19 (1786372399) [ 2315.517974] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:33:26 (1786372406) [ 2323.046659] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:33:33 (1786372413) [ 2329.893573] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:33:40 (1786372420) [ 2335.464914] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:33:46 (1786372426) [ 2341.564566] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:33:52 (1786372432) [ 2347.888946] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:33:58 (1786372438) [ 2354.931307] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:34:05 (1786372445) [ 2361.311126] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2115 sec ========== 10:34:12 (1786372452) [ 2361.568664] LNet: 144264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.574591] LNetError: 144264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.595761] LNet: Removed LNI 192.168.204.37@tcp [ 2362.121701] Key type .llcrypt unregistered [ 2362.123212] Key type ._llcrypt unregistered [ 2373.540852] Key type ._llcrypt registered [ 2373.542681] Key type .llcrypt registered [ 2373.886649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2373.895974] alg: No test for adler32 (adler32-zlib) [ 2375.021979] Lustre: Lustre: Build Version: 2.17.56_50_g6d2b241 [ 2375.283189] LNet: Added LNI 192.168.204.37@tcp [8/256/0/180] [ 2377.031142] Key type lgssc registered [ 2378.232130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.075347] Lustre: Mounted lustre-client - version 2.17.56_50_g6d2b241 [ 2438.578191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2444.910061] Lustre: Unmounted lustre-client [ 2464.078907] Key type lgssc unregistered [ 2464.299758] LNet: 147219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.307822] LNetError: 147219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.328956] LNet: Removed LNI 192.168.204.37@tcp [ 2464.970337] Key type .llcrypt unregistered [ 2464.973364] Key type ._llcrypt unregistered