[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 368338014 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002344] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007439] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009005] pid_max: default: 32768 minimum: 301 [ 0.010108] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013049] *** VALIDATE selinux *** [ 0.021513] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025323] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027067] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028058] *** VALIDATE tmpfs *** [ 0.029326] *** VALIDATE proc *** [ 0.030140] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032153] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033098] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035026] Spectre V2 : User space: Vulnerable [ 0.036003] Speculative Store Bypass: Vulnerable [ 0.038807] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.040120] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041377] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042009] ... version: 2 [ 0.042709] ... bit width: 48 [ 0.043005] ... generic registers: 4 [ 0.043727] ... value mask: 0000ffffffffffff [ 0.044005] ... max period: 00007fffffffffff [ 0.044994] ... fixed-purpose events: 3 [ 0.045005] ... event mask: 000000070000000f [ 0.046778] rcu: Hierarchical SRCU implementation. [ 0.047991] smp: Bringing up secondary CPUs ... [ 0.048319] x86: Booting SMP configuration: [ 0.049010] .... node #0, CPUs: #1 #2 #3 [ 0.051542] smp: Brought up 1 node, 4 CPUs [ 0.052737] smpboot: Max logical packages: 1 [ 0.053006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.268023] node 0 deferred pages initialised in 214ms [ 0.270500] devtmpfs: initialized [ 0.271154] x86/mm: Memory block size: 128MB [ 0.273110] gcov: version magic: 0x41383552 [ 0.274421] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.275058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.276158] pinctrl core: initialized pinctrl subsystem [ 0.277105] [ 0.277379] ************************************************************* [ 0.278009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279005] ** ** [ 0.280006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.281006] ** ** [ 0.282006] ** This means that this kernel is built to expose internal ** [ 0.283005] ** IOMMU data structures, which may compromise security on ** [ 0.284005] ** your system. ** [ 0.285006] ** ** [ 0.286006] ** If you see this message and you are not debugging the ** [ 0.287005] ** kernel, report this immediately to your vendor! ** [ 0.288005] ** ** [ 0.289005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290006] ************************************************************* [ 0.291398] NET: Registered protocol family 16 [ 0.292288] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.295272] cpuidle: using governor menu [ 0.296531] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.298352] PCI: Using configuration type 1 for base access [ 0.299100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.304116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.306100] cryptd: max_cpu_qlen set to 1000 [ 0.307170] ACPI: Added _OSI(Module Device) [ 0.308016] ACPI: Added _OSI(Processor Device) [ 0.308926] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.309010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.311699] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.316594] ACPI: Interpreter enabled [ 0.318053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319009] ACPI: Using IOAPIC for interrupt routing [ 0.321063] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.322233] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.328700] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330016] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.332011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.333028] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335765] acpiphp: Slot [2] registered [ 0.337073] acpiphp: Slot [3] registered [ 0.337805] acpiphp: Slot [4] registered [ 0.338050] acpiphp: Slot [5] registered [ 0.338835] acpiphp: Slot [6] registered [ 0.340047] acpiphp: Slot [7] registered [ 0.340748] acpiphp: Slot [8] registered [ 0.341041] acpiphp: Slot [9] registered [ 0.341809] acpiphp: Slot [10] registered [ 0.343053] acpiphp: Slot [11] registered [ 0.343812] acpiphp: Slot [12] registered [ 0.344041] acpiphp: Slot [13] registered [ 0.344800] acpiphp: Slot [14] registered [ 0.346044] acpiphp: Slot [15] registered [ 0.346800] acpiphp: Slot [16] registered [ 0.347041] acpiphp: Slot [17] registered [ 0.347811] acpiphp: Slot [18] registered [ 0.349043] acpiphp: Slot [19] registered [ 0.349773] acpiphp: Slot [20] registered [ 0.350044] acpiphp: Slot [21] registered [ 0.350862] acpiphp: Slot [22] registered [ 0.352042] acpiphp: Slot [23] registered [ 0.352819] acpiphp: Slot [24] registered [ 0.353063] acpiphp: Slot [25] registered [ 0.353827] acpiphp: Slot [26] registered [ 0.355042] acpiphp: Slot [27] registered [ 0.355794] acpiphp: Slot [28] registered [ 0.356042] acpiphp: Slot [29] registered [ 0.356792] acpiphp: Slot [30] registered [ 0.358064] acpiphp: Slot [31] registered [ 0.358802] PCI host bridge to bus 0000:00 [ 0.359008] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.360008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.361008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.363008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364008] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.366010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.367069] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.369483] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371491] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376367] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.378409] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383223] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386080] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388425] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391008] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.396705] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.400007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402675] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.406008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417914] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.422721] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424909] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435008] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.441462] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.443209] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.445241] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.447244] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.449122] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.454136] iommu: Default domain type: Passthrough [ 0.455290] SCSI subsystem initialized [ 0.456068] ACPI: bus type USB registered [ 0.456910] usbcore: registered new interface driver usbfs [ 0.458035] usbcore: registered new interface driver hub [ 0.459028] usbcore: registered new device driver usb [ 0.460069] pps_core: LinuxPPS API ver. 1 registered [ 0.461005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.462017] PTP clock support registered [ 0.463076] EDAC MC: Ver: 3.0.0 [ 0.464085] PCI: Using ACPI for IRQ routing [ 0.466500] NetLabel: Initializing [ 0.467010] NetLabel: domain hash size = 128 [ 0.468009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.470047] NetLabel: unlabeled traffic allowed by default [ 0.471068] vgaarb: loaded [ 0.472236] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.473008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.477262] clocksource: Switched to clocksource kvm-clock [ 0.554110] VFS: Disk quotas dquot_6.6.0 [ 0.554911] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.556230] *** VALIDATE ramfs *** [ 0.556840] *** VALIDATE hugetlbfs *** [ 0.558190] pnp: PnP ACPI init [ 0.559692] pnp: PnP ACPI: found 6 devices [ 0.571217] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.572925] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.573956] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.575070] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.576259] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.577412] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.578791] NET: Registered protocol family 2 [ 0.580124] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.583115] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.584882] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.588271] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.590026] TCP: Hash tables configured (established 65536 bind 65536) [ 0.591395] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.592974] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.594452] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.595897] NET: Registered protocol family 1 [ 0.597321] RPC: Registered named UNIX socket transport module. [ 0.598458] RPC: Registered udp transport module. [ 0.599265] RPC: Registered tcp transport module. [ 0.600055] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.601149] NET: Registered protocol family 44 [ 0.601991] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.603094] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.604141] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.605232] PCI: CLS 0 bytes, default 64 [ 0.606113] Unpacking initramfs... [ 1.732081] debug: unmapping init [mem 0xffff97d13cc64000-0xffff97d13ffcffff] [ 1.734732] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.736614] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.739679] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.149862] Initialise system trusted keyrings [ 2.150780] Key type blacklist registered [ 2.151794] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.157680] zbud: loaded [ 2.159479] *** VALIDATE nfs *** [ 2.160151] *** VALIDATE nfs4 *** [ 2.161023] pstore: using deflate compression [ 2.162754] Platform Keyring initialized [ 2.225683] NET: Registered protocol family 38 [ 2.226645] Key type asymmetric registered [ 2.227448] Asymmetric key parser 'x509' registered [ 2.228392] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.230070] io scheduler mq-deadline registered [ 2.230925] io scheduler kyber registered [ 2.231931] io scheduler bfq registered [ 2.232945] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.234461] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.236034] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.237503] ACPI: Power Button [PWRF] [ 2.319931] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.384345] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.469839] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.497047] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.522240] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.525269] Non-volatile memory driver v1.3 [ 2.525955] Linux agpgart interface v0.103 [ 2.545860] virtio_blk virtio1: [vda] 132944 512-byte logical blocks (68.1 MB/64.9 MiB) [ 2.547435] vda: detected capacity change from 0 to 68067328 [ 2.557268] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.558717] vdb: detected capacity change from 0 to 1073741824 [ 2.562097] libphy: Fixed MDIO Bus: probed [ 2.564889] usbcore: registered new interface driver usbserial_generic [ 2.566132] usbserial: USB Serial support registered for generic [ 2.567269] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.569408] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.570324] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.571543] mousedev: PS/2 mouse device common for all mice [ 2.572954] rtc_cmos 00:05: RTC can wake from S4 [ 2.574453] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.576441] rtc_cmos 00:05: registered as rtc0 [ 2.577488] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.577875] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.580156] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.580859] intel_pstate: CPU model not supported [ 2.584021] hid: raw HID events driver (C) Jiri Kosina [ 2.585038] usbcore: registered new interface driver usbhid [ 2.586043] usbhid: USB HID core driver [ 2.586834] drop_monitor: Initializing network drop monitor service [ 2.588063] Initializing XFRM netlink socket [ 2.589118] NET: Registered protocol family 10 [ 2.590777] Segment Routing with IPv6 [ 2.591511] NET: Registered protocol family 17 [ 2.592554] mpls_gso: MPLS GSO support [ 2.597130] RAS: Correctable Errors collector initialized. [ 2.598158] AVX version of gcm_enc/dec engaged. [ 2.598989] AES CTR mode by8 optimization enabled [ 2.640412] sched_clock: Marking stable (2640383906, 0)->(3338412091, -698028185) [ 2.642822] registered taskstats version 1 [ 2.643731] Loading compiled-in X.509 certificates [ 2.644557] zswap: loaded using pool lzo/zbud [ 2.660992] Key type big_key registered [ 2.667559] Key type encrypted registered [ 2.668212] ima: No TPM chip found, activating TPM-bypass! [ 2.669040] ima: Allocated hash algorithm: sha1 [ 2.669707] ima: No architecture policies found [ 2.670417] evm: Initialising EVM extended attributes: [ 2.671173] evm: security.selinux [ 2.671663] evm: security.ima [ 2.672114] evm: security.capability [ 2.672629] evm: HMAC attrs: 0x1 [ 2.673785] rtc_cmos 00:05: setting system clock to 2025-08-27 16:16:29 UTC (1756311389) [ 2.677133] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 2.678369] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 2.685107] Write protecting the kernel read-only data: 28672k [ 2.686717] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 2.687907] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 2.707490] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.711085] systemd[1]: Detected virtualization kvm. [ 2.711823] systemd[1]: Detected architecture x86-64. [ 2.712675] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.733518] systemd[1]: No hostname configured. [ 2.734313] systemd[1]: Set hostname to . [ 2.735323] random: systemd: uninitialized urandom read (16 bytes read) [ 2.736677] systemd[1]: Initializing machine ID from random generator. [ 2.777571] random: ln: uninitialized urandom read (6 bytes read) [ 2.836681] random: systemd: uninitialized urandom read (16 bytes read) [ 2.837880] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 2.840210] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 2.842570] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.289985] device-mapper: uevent: version 1.0.3 [ 3.291213] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 3.863409] virtio_net virtio0 ens2: renamed from eth0 [ 3.922761] scsi host0: ata_piix [ 3.927347] scsi host1: ata_piix [ 3.928560] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 3.929958] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.176254] random: fast init done [ 7.921053] dracut-initqueue[592]: RTNETLINK answers: File exists [ 8.175813] random: crng init done [ 8.176431] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 8.447924] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.204705] printk: systemd: 23 output lines suppressed due to ratelimiting [ 9.324597] SELinux: Disabled at runtime. [ 9.369805] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.373407] systemd[1]: Detected virtualization kvm. [ 9.374273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 9.634522] systemd[1]: initrd-switch-root.service: Succeeded. [ 9.636133] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 9.638580] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 9.640362] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 9.641760] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 9.645994] systemd[1]: Starting Journal Service... Starting Journal Service... [ 9.648066] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ 9.674284] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 10.026591] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 10.256211] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.308457] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.375988] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.382481] EDAC sbridge: Ver: 1.1.2 [ 11.283673] Key type dns_resolver registered [ 11.507246] NFS: Registering the id_resolver key type [ 11.508166] Key type id_resolver registered [ 11.508929] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-client login: [ 34.420096] libcfs: loading out-of-tree module taints kernel. [ 34.452711] Key type ._llcrypt registered [ 34.469088] Key type .llcrypt registered [ 34.728045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 34.732868] alg: No test for adler32 (adler32-zlib) [ 35.714738] Lustre: Lustre: Build Version: 2.16.57_134_ga204b13 [ 36.046842] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 37.655136] Key type lgssc registered [ 38.208558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.709655] Lustre: Mounted lustre-client [ 88.331192] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 98.578918] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing check_logdir /tmp/testlogs/ [ 99.800931] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing yml_node [ 101.038152] Lustre: DEBUG MARKER: Client: 2.16.57.134 [ 101.795531] Lustre: DEBUG MARKER: MDS: 2.16.57.134 [ 102.522781] Lustre: DEBUG MARKER: OSS: 2.16.57.134 [ 103.007323] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 27 12:18:09 EDT 2025 [ 107.264501] LustreError: 7896:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff97d18549f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 107.271430] LustreError: 7896:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 107.284148] Lustre: Unmounted lustre-client [ 118.808247] Key type lgssc unregistered [ 118.978864] LNet: 8342:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 118.981503] LNetError: 8342:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 118.991992] LNet: Removed LNI 192.168.204.13@tcp [ 119.333126] Key type .llcrypt unregistered [ 119.334178] Key type ._llcrypt unregistered [ 126.960814] Key type ._llcrypt registered [ 126.961960] Key type .llcrypt registered [ 127.021415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 127.026162] alg: No test for adler32 (adler32-zlib) [ 127.820662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 127.842712] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 127.848949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 129.020682] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 129.045560] LNet: 9206:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 129.047744] LNetError: 9206:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 129.053385] LNet: Removed LNI 192.168.204.13@tcp [ 129.266236] Key type .llcrypt unregistered [ 129.267197] Key type ._llcrypt unregistered [ 131.012135] Key type ._llcrypt registered [ 131.014088] Key type .llcrypt registered [ 131.390197] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:18:37 (1756311517) [ 131.552924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.557160] alg: No test for adler32 (adler32-zlib) [ 132.347687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 132.369776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 132.375653] LNet: 9618:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 132.381708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 132.396073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 132.401570] LNet: 9625:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 134.149283] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:18:40 (1756311520) [ 134.173392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 134.180568] LNet: 10197:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 134.188050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 134.212378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 135.914675] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:18:42 (1756311522) [ 135.937867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 135.944613] LNet: 10781:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 135.952778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 135.977431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 138.839883] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:18:45 (1756311525) [ 138.887562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 138.901934] LNet: 11364:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 138.918067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 138.960534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 141.310571] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:18:47 (1756311527) [ 141.334771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 141.348977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 141.374451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 141.482946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 143.717678] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:18:50 (1756311530) [ 143.744688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 143.752850] LNet: 12537:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 143.754928] LNet: 12537:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 143.765627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 143.791529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 143.903556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 144.032540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 146.298144] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:18:52 (1756311532) [ 146.331985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.350500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.388216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 148.790223] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:18:55 (1756311535) [ 148.826648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 148.850611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 148.885323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 148.904641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 149.026804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 149.046756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 149.179408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 149.190666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 149.301033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 149.313146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 149.438965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 149.454832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 149.594357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 149.606516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 149.721753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 149.733725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 149.873749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 149.884365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 149.994984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 150.005653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 150.117517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 150.129682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 150.256504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 150.273760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 150.410731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 150.426698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 150.566322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 150.577767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 152.779363] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:18:59 (1756311539) [ 152.811408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 152.821276] LNet: 14395:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 152.823412] LNet: 14395:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 152.833054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 152.868660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 152.880770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 155.066083] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:19:01 (1756311541) [ 155.100697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 155.117862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 155.152155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 155.164171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 157.371872] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:19:03 (1756311543) [ 157.402839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 157.423773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 157.466323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 157.482123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 159.677916] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:19:06 (1756311546) [ 159.709392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 159.725578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 159.756906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 159.767865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 161.933087] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:19:08 (1756311548) [ 161.964659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.983431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 162.020210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 162.032399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 164.657328] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:19:10 (1756311550) [ 164.697975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 164.722501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 164.766424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 164.780988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 167.318913] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:19:13 (1756311553) [ 167.357534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.377944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.423620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 167.557716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 170.263046] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:19:16 (1756311556) [ 170.309916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.326392] LNet: 18503:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 170.329968] LNet: 18503:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 170.345972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.429651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 170.533415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 172.592481] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:19:19 (1756311559) [ 172.622270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.637303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.687172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 172.759146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 175.516615] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:19:21 (1756311561) [ 175.565191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 175.594394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 175.813932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 175.928404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 178.657924] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:19:24 (1756311564) [ 178.707230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.730698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.773238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 178.801949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 181.562977] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:19:27 (1756311567) [ 181.611495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.635743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.676281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 181.695607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 184.501568] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:19:30 (1756311570) [ 184.542335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.567296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.608668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 187.548165] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:19:33 (1756311573) [ 187.598131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.627849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.679997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 190.547778] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:19:36 (1756311576) [ 190.582977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.605742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.652590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 190.784582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 193.604253] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:19:39 (1756311579) [ 193.652624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.677927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.720095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 193.735585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 196.606202] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:19:42 (1756311582) [ 196.640522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.657829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.694447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 196.710829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 199.368448] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:19:45 (1756311585) [ 199.416484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.443619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.487390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 199.505364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 202.153854] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:19:48 (1756311588) [ 202.205585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.235001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.273542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 202.290366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 204.801123] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:19:51 (1756311591) [ 204.848228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.862064] LNet: 25687:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 204.865677] LNet: 25687:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 204.882418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.915798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 204.946617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 204.956336] LNetError: 25704:0:(peer.c:2202:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 204.976292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 204.993453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 207.787553] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:19:53 (1756311593) [ 207.836978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.862512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.896622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jK0P5bEIBs [ 207.927844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jK0P5bEIBs [ 207.947938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jK0P5bEIBs [ 210.531508] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:19:56 (1756311596) [ 210.570640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.590503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.627699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 210.644357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 210.671120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 213.344974] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:19:59 (1756311599) [ 213.400624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.428479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.468653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 214.442259] LNet: Message forwarding will be enabled in 2s [ 217.503092] LNet: Message forwarding enabled [ 217.606835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 217.616305] LNet: Message forwarding will stop in 2s to 6s [ 221.830599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 222.195596] LNet: Message forwarding disabled [ 222.771036] LNet: Message forwarding will be enabled in 2s [ 225.449041] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:20:11 (1756311611) [ 225.495691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.039332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.068223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 228.635274] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:20:14 (1756311614) [ 228.673298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 228.693498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 228.719401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 228.730316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 228.742185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 228.753674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 228.763780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 228.774921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 228.786620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 228.800766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 228.815527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 228.826394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 228.837210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 228.847961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 228.860383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 228.872137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 228.885899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 228.897514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 228.910112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 228.921173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 228.931950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 228.939506] LNetError: 28770:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 231.245306] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:20:17 (1756311617) [ 231.292742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.326238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.359169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 234.204334] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:20:20 (1756311620) [ 235.056933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg413-server [ 235.909693] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:20:22 (1756311622) [ 235.954625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.972854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.996898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 236.006348] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 236.726592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[14-18]@tcp [ 236.738251] LNet: 30481:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 239.207782] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:20:25 (1756311625) [ 239.839420] LNetError: 30280:0:(lib-lnet.h:1309:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.16@tcp has gone from up to down [ 239.993496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg413-server [ 240.796225] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:20:26 (1756311626) [ 240.848041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.935158] LNetError: 30273:0:(lib-lnet.h:1309:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.18@tcp has gone from up to down [ 243.938907] LNetError: 30273:0:(lib-lnet.h:1309:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 243.942422] LNetError: 31411:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 243.952900] LNet: Removed LNI 192.168.204.13@tcp [ 243.964099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.986517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 243.998283] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 244.197036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[14-19/2]@tcp [ 244.209916] LNet: 31531:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 244.214806] LNet: 31531:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 244.224829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[14-19/2]@tcp [ 246.772947] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:20:32 (1756311632) [ 246.817386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.830471] LNetError: 32117:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 246.840761] LNet: Removed LNI 192.168.204.13@tcp [ 246.850681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.891845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 246.926868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 246.945779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.974442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 246.997545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.012945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.037055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 247.078902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.099420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.131337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.161449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.174636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.200444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 247.243156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.263802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.299827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.342993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.365601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.397423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 247.448312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.467514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.494595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.519730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.530146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.544716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 247.568629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.582836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.612493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.649683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.666619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.692210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 247.718675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.735171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.763674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 250.402343] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:20:36 (1756311636) [ 251.247407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg413-server [ 252.043250] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:20:38 (1756311638) [ 252.909667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg413-server [ 253.688831] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:20:39 (1756311639) [ 254.014927] Key type .llcrypt unregistered [ 254.017730] Key type ._llcrypt unregistered [ 255.642193] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 255.707735] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 255.709902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 255.711390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 257.941329] Key type ._llcrypt registered [ 257.943957] Key type .llcrypt registered [ 258.039855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 258.046311] alg: No test for adler32 (adler32-zlib) [ 258.858210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.886801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 258.895428] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 259.215073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 259.222773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 260.105729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 260.115551] LNet: Removed LNI 10.1.2.3@tcp [ 260.164708] LNet: 34231:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.167274] LNetError: 34231:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 260.176381] LNet: Removed LNI 192.168.204.13@tcp [ 260.374643] Key type .llcrypt unregistered [ 260.375539] Key type ._llcrypt unregistered [ 263.709524] Key type ._llcrypt registered [ 263.710775] Key type .llcrypt registered [ 264.301502] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:20:50 (1756311650) [ 264.495376] Key type .llcrypt unregistered [ 264.496516] Key type ._llcrypt unregistered [ 266.475084] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 266.558763] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 266.560238] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 266.561601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 268.779410] Key type ._llcrypt registered [ 268.780969] Key type .llcrypt registered [ 268.873806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 268.882353] alg: No test for adler32 (adler32-zlib) [ 269.685129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.708263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 269.717760] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 270.023141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 270.036025] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 271.132928] LNet: 35703:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.134966] LNetError: 35703:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 271.136598] LNet: Removed LNI 192.168.204.13@tcp [ 271.141305] LNet: Removed LNI 10.1.2.3@tcp [ 271.388435] Key type .llcrypt unregistered [ 271.390316] Key type ._llcrypt unregistered [ 275.332100] Key type ._llcrypt registered [ 275.333380] Key type .llcrypt registered [ 275.916582] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:21:02 (1756311662) [ 276.132348] Key type .llcrypt unregistered [ 276.133551] Key type ._llcrypt unregistered [ 277.761290] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 277.812456] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 277.813697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 277.814916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 280.068252] Key type ._llcrypt registered [ 280.070327] Key type .llcrypt registered [ 280.174222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 280.180872] alg: No test for adler32 (adler32-zlib) [ 280.989827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.010591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 281.019302] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 281.812486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 281.821051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 282.467943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 282.476103] LNet: Removed LNI 10.1.2.3@tcp [ 282.482726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 282.497666] LNet: Removed LNI 10.1.2.31@tcp [ 282.505656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 282.518763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 282.902962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 282.914166] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 283.416887] LNet: 37599:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.420300] LNetError: 37599:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 283.423520] LNet: Removed LNI 10.1.2.3@tcp [ 283.435643] LNet: Removed LNI 10.1.2.31@tcp [ 283.730325] Key type .llcrypt unregistered [ 283.731178] Key type ._llcrypt unregistered [ 287.512178] Key type ._llcrypt registered [ 287.513781] Key type .llcrypt registered [ 288.145664] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:21:14 (1756311674) [ 288.297860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 288.302775] alg: No test for adler32 (adler32-zlib) [ 289.111193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.132461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 289.138931] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 289.522652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.534412] LNet: 38542:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.538374] LNetError: 38542:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.549519] LNet: Removed LNI 192.168.204.13@tcp [ 289.561597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.596326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.13@tcp [ 289.610325] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 290.011214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.13@tcp [ 290.035894] LNet: Removed LNI 192.168.204.13@tcp [ 290.072965] LNet: 38674:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.415587] Key type .llcrypt unregistered [ 290.417490] Key type ._llcrypt unregistered [ 293.787977] Key type ._llcrypt registered [ 293.789053] Key type .llcrypt registered [ 294.154481] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:21:20 (1756311680) [ 294.778524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg413-server [ 295.352616] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:21:21 (1756311681) [ 295.579275] Key type .llcrypt unregistered [ 295.580186] Key type ._llcrypt unregistered [ 297.181405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 297.247662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 297.250259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 297.252187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 299.484365] Key type ._llcrypt registered [ 299.486108] Key type .llcrypt registered [ 299.552238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 299.558270] alg: No test for adler32 (adler32-zlib) [ 300.384789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.418387] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 300.795825] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 301.859253] LNet: 40494:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.863423] LNetError: 40494:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 301.867820] LNet: Removed LNI 192.168.204.13@tcp1 [ 301.880569] LNet: Removed LNI 10.1.2.3@tcp2 [ 302.225464] Key type .llcrypt unregistered [ 302.227362] Key type ._llcrypt unregistered [ 305.485895] Key type ._llcrypt registered [ 305.486865] Key type .llcrypt registered [ 305.852178] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:21:32 (1756311692) [ 306.067421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 306.072444] alg: No test for adler32 (adler32-zlib) [ 306.868829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.886665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 306.894379] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 307.177125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.185330] LNet: 41450:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.187960] LNetError: 41450:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.197365] LNet: Removed LNI 192.168.204.13@tcp [ 307.205822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.246660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 307.256561] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 309.627850] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:21:35 (1756311695) [ 309.658620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.668584] LNet: 42040:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.670439] LNetError: 42040:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.678620] LNet: Removed LNI 192.168.204.13@tcp [ 309.686401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.708818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 309.717331] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 309.937251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.13 [ 309.943622] LNet: Removed LNI 192.168.204.13@tcp [ 312.722413] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:21:38 (1756311698) [ 312.772788] LNet: 42738:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.777226] LNetError: 42738:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.785791] LNet: Removed LNI 192.168.204.13@tcp [ 313.096632] Key type .llcrypt unregistered [ 313.098762] Key type ._llcrypt unregistered [ 314.825308] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 314.924357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 314.926235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 314.928302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 317.279716] Key type ._llcrypt registered [ 317.281608] Key type .llcrypt registered [ 317.376372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 317.385465] alg: No test for adler32 (adler32-zlib) [ 318.371629] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 321.452686] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:21:47 (1756311707) [ 321.507744] LNet: 43637:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.512742] LNetError: 43637:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 321.523590] LNet: Removed LNI 10.1.2.3@tcp [ 321.854721] Key type .llcrypt unregistered [ 321.856313] Key type ._llcrypt unregistered [ 323.499433] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 323.616253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 323.619153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 323.621836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 325.952002] Key type ._llcrypt registered [ 325.953478] Key type .llcrypt registered [ 326.024372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 326.028839] alg: No test for adler32 (adler32-zlib) [ 326.843099] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 329.529108] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:21:55 (1756311715) [ 329.581894] LNet: 44525:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.585925] LNetError: 44525:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 329.593313] LNet: Removed LNI 10.1.2.3@tcp [ 329.920485] Key type .llcrypt unregistered [ 329.923124] Key type ._llcrypt unregistered [ 331.590798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 331.689114] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 331.691771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 331.694616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 334.011634] Key type ._llcrypt registered [ 334.013492] Key type .llcrypt registered [ 334.072923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 334.078776] alg: No test for adler32 (adler32-zlib) [ 334.915186] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 337.607335] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:22:03 (1756311723) [ 337.651320] LNet: 45415:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.655218] LNetError: 45415:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 337.663760] LNet: Removed LNI 10.1.2.3@tcp [ 337.932436] Key type .llcrypt unregistered [ 337.934760] Key type ._llcrypt unregistered [ 339.718653] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.807866] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.809307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.810830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 342.117960] Key type ._llcrypt registered [ 342.120129] Key type .llcrypt registered [ 342.226187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 342.236889] alg: No test for adler32 (adler32-zlib) [ 343.038219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.069313] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 347.022181] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:22:13 (1756311733) [ 347.217129] LNet: 46641:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.220744] LNetError: 46641:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 347.229833] LNet: Removed LNI 10.1.2.3@tcp [ 347.527579] Key type .llcrypt unregistered [ 347.529641] Key type ._llcrypt unregistered [ 349.170833] Key type ._llcrypt registered [ 349.171913] Key type .llcrypt registered [ 349.246123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 349.250650] alg: No test for adler32 (adler32-zlib) [ 350.050449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 350.060855] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 353.388754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 353.736924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_interrupt -r 1 [ 353.755697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_interrupt -r 1 [ 353.775564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 353.786319] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 353.924568] LNet: 47065:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 353.957818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 353.973247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 353.988978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 354.236748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_dropped -r 1 [ 354.253724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_dropped -r 1 [ 354.272591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 354.429955] LNet: 47162:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 354.456834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 354.470425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 354.485399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 354.748522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_aborted -r 1 [ 354.765098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_aborted -r 1 [ 354.785788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 354.797404] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 354.799990] LNet: Skipped 1 previous similar message [ 354.977131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 354.994282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 355.011513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 355.319209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_no_route -r 1 [ 355.339529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_no_route -r 1 [ 355.358077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 355.501470] LNet: 47357:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 355.505141] LNet: 47357:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 355.532756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 355.545785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 355.559242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 355.769618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_timeout -r 1 [ 355.784572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_timeout -r 1 [ 355.800276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 355.808958] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 355.812330] LNet: Skipped 1 previous similar message [ 355.963324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 355.975482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 355.986925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 356.237762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_error -r 1 [ 356.256481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_error -r 1 [ 356.273674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 356.450420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 356.469477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 356.486895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 356.588366] LNet: 47587:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.590236] LNetError: 47587:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.597349] LNet: Removed LNI 192.168.204.13@tcp [ 356.753365] Key type .llcrypt unregistered [ 356.754480] Key type ._llcrypt unregistered [ 362.487769] Key type ._llcrypt registered [ 362.489085] Key type .llcrypt registered [ 362.907697] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:22:29 (1756311749) [ 363.287251] Key type .llcrypt unregistered [ 363.289096] Key type ._llcrypt unregistered [ 365.079673] Key type ._llcrypt registered [ 365.081720] Key type .llcrypt registered [ 365.179422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 365.187150] alg: No test for adler32 (adler32-zlib) [ 366.001774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 366.013926] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 369.300551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 373.359262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.375321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 373.387194] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 374.018579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_interrupt -r 1 [ 374.030445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 1 [ 374.041552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_interrupt -r 1 [ 374.052516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 1 [ 374.062871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_interrupt -r 1 [ 374.073606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 1 [ 374.084382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_interrupt -r 1 [ 374.096147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 1 [ 374.108348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 374.114198] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 374.116968] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 374.243542] LNet: 49031:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 374.279370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 374.293803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 374.307643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 374.514533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_dropped -r 1 [ 374.531307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 1 [ 374.547540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_dropped -r 1 [ 374.563725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 1 [ 374.581535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_dropped -r 1 [ 374.602426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 1 [ 374.621815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_dropped -r 1 [ 374.638994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 1 [ 374.658410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 374.670175] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 374.674264] LNet: Skipped 2 previous similar messages [ 374.774407] LNet: 49141:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 374.797805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 374.808678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 374.819308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 375.006227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_aborted -r 1 [ 375.017780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 1 [ 375.028695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_aborted -r 1 [ 375.038992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 1 [ 375.050357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_aborted -r 1 [ 375.061356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 1 [ 375.072120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_aborted -r 1 [ 375.082438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 1 [ 375.093177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 375.225472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.235310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.245620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.440508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_no_route -r 1 [ 375.451309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 1 [ 375.462306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_no_route -r 1 [ 375.472562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 1 [ 375.485329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_no_route -r 1 [ 375.500642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 1 [ 375.516767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_no_route -r 1 [ 375.530174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 1 [ 375.542051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 375.654611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.664216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.673241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.820896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_timeout -r 1 [ 375.831593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 1 [ 375.842204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_timeout -r 1 [ 375.852536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 1 [ 375.863420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_timeout -r 1 [ 375.873665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 1 [ 375.883933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_timeout -r 1 [ 375.894212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 1 [ 375.904725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 375.911769] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 375.914345] LNet: Skipped 9 previous similar messages [ 376.003608] LNet: 49468:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 376.005626] LNet: 49468:0:(api-ni.c:337:recovery_interval_set()) Skipped 2 previous similar messages [ 376.024912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.034473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.045747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 376.203517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_error -r 1 [ 376.214549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_error -r 1 [ 376.224455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_error -r 1 [ 376.235631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_error -r 1 [ 376.245782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_error -r 1 [ 376.256491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_error -r 1 [ 376.269692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_error -r 1 [ 376.280260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_error -r 1 [ 376.291189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 376.402110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 376.411893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 376.422235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 378.689383] LNet: 49647:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.695495] LNetError: 49647:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.701966] LNet: Removed LNI 192.168.204.13@tcp [ 378.711354] LNet: Removed LNI 192.168.204.13@tcp1 [ 378.890663] Key type .llcrypt unregistered [ 378.891865] Key type ._llcrypt unregistered [ 385.063837] Key type ._llcrypt registered [ 385.066108] Key type .llcrypt registered [ 385.663890] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:22:51 (1756311771) [ 386.025442] Key type .llcrypt unregistered [ 386.027409] Key type ._llcrypt unregistered [ 388.239841] Key type ._llcrypt registered [ 388.242900] Key type .llcrypt registered [ 388.344575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 388.354671] alg: No test for adler32 (adler32-zlib) [ 389.163346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 389.177352] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 392.414836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 392.766436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_dropped -r 1 [ 392.786408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_dropped -r 1 [ 392.805865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 392.816208] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 392.958153] LNet: 50881:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 392.990855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 393.007931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 393.024296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 393.284205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_error -r 1 [ 393.302887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_error -r 1 [ 393.320867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 393.331931] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 393.489268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 393.506126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 393.524244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 393.813993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_timeout -r 1 [ 393.832946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_timeout -r 1 [ 393.854933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 394.001060] LNet: 51075:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 394.003645] LNet: 51075:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 394.038619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 394.057727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 394.078650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 394.220351] LNet: 51111:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.224270] LNetError: 51111:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 394.237575] LNet: Removed LNI 192.168.204.13@tcp [ 394.455504] Key type .llcrypt unregistered [ 394.457428] Key type ._llcrypt unregistered [ 400.814575] Key type ._llcrypt registered [ 400.815566] Key type .llcrypt registered [ 401.410603] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:23:07 (1756311787) [ 401.836771] Key type .llcrypt unregistered [ 401.838498] Key type ._llcrypt unregistered [ 404.197596] Key type ._llcrypt registered [ 404.198535] Key type .llcrypt registered [ 404.246910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 404.250981] alg: No test for adler32 (adler32-zlib) [ 405.062090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 405.073616] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 408.772558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 411.395190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.416280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 411.429728] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 412.057759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_dropped -r 1 [ 412.071106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 1 [ 412.085536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_dropped -r 1 [ 412.099919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 1 [ 412.112835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_dropped -r 1 [ 412.127310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 1 [ 412.141626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_dropped -r 1 [ 412.156464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 1 [ 412.169980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 412.178561] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 412.181404] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 412.285987] LNet: 52557:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.307506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.319641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 412.332714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 412.346549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.558532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_error -r 1 [ 412.573287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_error -r 1 [ 412.586745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_error -r 1 [ 412.601706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_error -r 1 [ 412.616460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_error -r 1 [ 412.631454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_error -r 1 [ 412.647315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_error -r 1 [ 412.663646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_error -r 1 [ 412.679610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 412.692089] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 412.697332] LNet: Skipped 2 previous similar messages [ 412.828913] LNet: 52668:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.849494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 412.859511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 412.869556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 412.879558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 413.064311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_timeout -r 1 [ 413.079533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_timeout -r 1 [ 413.095394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_timeout -r 1 [ 413.111749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_timeout -r 1 [ 413.130441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_timeout -r 1 [ 413.148566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_timeout -r 1 [ 413.164933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_timeout -r 1 [ 413.180294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_timeout -r 1 [ 413.199250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.113@tcp [ 413.372043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 413.389197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 413.407687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 413.429537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 414.947634] LNet: 52851:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.952246] LNetError: 52851:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 414.957254] LNet: Removed LNI 192.168.204.13@tcp [ 414.967602] LNet: Removed LNI 192.168.204.13@tcp1 [ 415.186644] Key type .llcrypt unregistered [ 415.188897] Key type ._llcrypt unregistered [ 421.133865] Key type ._llcrypt registered [ 421.135247] Key type .llcrypt registered [ 421.806671] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:23:27 (1756311807) [ 422.044338] Key type .llcrypt unregistered [ 422.046352] Key type ._llcrypt unregistered [ 424.183454] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 424.238891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 424.243400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 424.244928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 426.557945] Key type ._llcrypt registered [ 426.560055] Key type .llcrypt registered [ 426.654810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 426.664217] alg: No test for adler32 (adler32-zlib) [ 427.483977] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 427.503859] LNet: 53937:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.509395] LNetError: 53937:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.518556] LNet: Removed LNI 192.168.204.13@tcp [ 427.913979] Key type .llcrypt unregistered [ 427.915867] Key type ._llcrypt unregistered [ 430.096097] Key type ._llcrypt registered [ 430.099226] Key type .llcrypt registered [ 430.179772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 430.186410] alg: No test for adler32 (adler32-zlib) [ 431.009539] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 431.012134] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 431.034652] LNet: 54206:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.039489] LNetError: 54206:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 431.043803] LNet: Removed LNI 192.168.204.13@tcp [ 431.054681] LNet: Removed LNI 10.1.2.3@tcp1 [ 431.439468] Key type .llcrypt unregistered [ 431.441521] Key type ._llcrypt unregistered [ 433.645578] Key type ._llcrypt registered [ 433.647944] Key type .llcrypt registered [ 433.736354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 433.744707] alg: No test for adler32 (adler32-zlib) [ 434.554711] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 434.557147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 434.573159] LNet: 54474:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.575776] LNetError: 54474:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.578421] LNet: Removed LNI 192.168.204.13@tcp [ 434.583384] LNet: Removed LNI 10.1.2.3@tcp [ 434.933610] Key type .llcrypt unregistered [ 434.935666] Key type ._llcrypt unregistered [ 437.161569] Key type ._llcrypt registered [ 437.164103] Key type .llcrypt registered [ 437.260693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 437.268671] alg: No test for adler32 (adler32-zlib) [ 438.088781] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 438.090097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 438.101856] LNet: 54744:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.104327] LNetError: 54744:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.106110] LNet: Removed LNI 192.168.204.13@tcp [ 438.112289] LNet: Removed LNI 10.1.2.3@tcp [ 438.386360] Key type .llcrypt unregistered [ 438.387855] Key type ._llcrypt unregistered [ 440.532141] Key type ._llcrypt registered [ 440.533873] Key type .llcrypt registered [ 440.632434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 440.638950] alg: No test for adler32 (adler32-zlib) [ 441.458308] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 441.461541] LNetError: 55004:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 441.466735] LNetError: Error -100 starting up LNI tcp [ 441.475509] LNet: Removed LNI 192.168.204.13@tcp [ 441.849639] Key type .llcrypt unregistered [ 441.852211] Key type ._llcrypt unregistered [ 445.726670] Key type ._llcrypt registered [ 445.727600] Key type .llcrypt registered [ 446.226228] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:23:52 (1756311832) [ 446.646288] Key type .llcrypt unregistered [ 446.648308] Key type ._llcrypt unregistered [ 448.782360] Key type ._llcrypt registered [ 448.784836] Key type .llcrypt registered [ 448.882846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 448.889943] alg: No test for adler32 (adler32-zlib) [ 449.702271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 449.714670] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 453.342561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 453.708210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e network_timeout -r 1 [ 453.726757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e network_timeout -r 1 [ 453.743096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 453.753288] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 453.932329] LNet: 56242:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 453.967255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 453.984663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 454.130633] LNet: 56276:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.138906] LNetError: 56276:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.153573] LNet: Removed LNI 192.168.204.13@tcp [ 454.379587] Key type .llcrypt unregistered [ 454.381317] Key type ._llcrypt unregistered [ 460.568609] Key type ._llcrypt registered [ 460.570503] Key type .llcrypt registered [ 460.676115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 460.683683] alg: No test for adler32 (adler32-zlib) [ 461.494234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 461.505872] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 465.100060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 468.710234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.731448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 468.744744] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 469.471227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e network_timeout -r 1 [ 469.488878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e network_timeout -r 1 [ 469.504945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e network_timeout -r 1 [ 469.520286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e network_timeout -r 1 [ 469.534887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e network_timeout -r 1 [ 469.553865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e network_timeout -r 1 [ 469.577586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e network_timeout -r 1 [ 469.598978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e network_timeout -r 1 [ 469.620598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 469.631667] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 469.727143] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 469.729752] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 469.784373] LNet: 57134:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 469.808256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 469.821462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 472.235523] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 472.385592] LNet: 57203:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.389829] LNetError: 57203:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.394376] LNet: Removed LNI 192.168.204.13@tcp [ 472.403590] LNet: Removed LNI 192.168.204.13@tcp1 [ 472.609670] Key type .llcrypt unregistered [ 472.611682] Key type ._llcrypt unregistered [ 479.154668] Key type ._llcrypt registered [ 479.156147] Key type .llcrypt registered [ 479.807454] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:24:25 (1756311865) [ 479.986129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 479.992970] alg: No test for adler32 (adler32-zlib) [ 480.798634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.819124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 480.828510] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 481.221599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 481.235848] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 481.692432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 481.735153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 481.763316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 481.785222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 481.807135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 481.827650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 481.847457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.13@tcp [ 481.860810] LNetError: 58292:0:(api-ni.c:4229:lnet_ni_set_healthv()) manually adding local NI 192.168.204.13@tcp to recovery [ 482.847394] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 482.847465] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 484.896394] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 485.117985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 488.992263] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 489.424175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 489.446188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 489.480611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.492638] LNet: 58409:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.497250] LNetError: 58409:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.501897] LNet: Removed LNI 192.168.204.13@tcp [ 489.510488] LNet: Removed LNI 192.168.204.13@tcp1 [ 489.519929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.542970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.553259] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 489.848120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 489.859899] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 490.282541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 490.306791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 490.342823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 490.370821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 490.391639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 490.410356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 490.431125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 490.451568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.13@tcp [ 490.466211] LNetError: 58675:0:(api-ni.c:4229:lnet_ni_set_healthv()) manually adding local NI 192.168.204.13@tcp to recovery [ 492.640374] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 492.644274] LNet: Skipped 1 previous similar message [ 493.713821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 496.736165] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 501.184784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 501.204682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 501.223865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 504.464735] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:24:50 (1756311890) [ 504.535110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.549623] LNet: 59390:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.552883] LNetError: 59390:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.556065] LNetError: 59390:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 504.558964] LNet: Removed LNI 192.168.204.13@tcp [ 504.565927] LNet: Removed LNI 192.168.204.13@tcp1 [ 504.576651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.603552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 504.615766] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 504.998976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 505.011502] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 505.466183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 505.504478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 505.522198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 505.542508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 505.563141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 505.581735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 505.598591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.13@tcp --health 0 [ 505.631177] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.13@tcp [ 505.648530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 506.655457] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 506.658826] LNet: Skipped 1 previous similar message [ 521.292786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 521.348346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 521.357791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 521.367169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.13@tcp --health 500 [ 521.397490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 527.682643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 527.725330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.738736] LNet: 59885:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.743822] LNetError: 59885:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.748033] LNetError: 59885:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 527.751476] LNet: Removed LNI 192.168.204.13@tcp [ 527.759452] LNet: Removed LNI 192.168.204.13@tcp1 [ 527.767736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.795160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 527.809705] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 528.195797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 528.208520] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 528.649478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 528.671509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 528.708758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 528.729334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 528.749307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 528.767343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 528.782935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 528.796653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.13@tcp --health 0 [ 529.823409] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 529.827366] LNet: Skipped 3 previous similar messages [ 529.954901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 539.547935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 539.566991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 539.586926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.770585] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:25:28 (1756311928) [ 542.978899] LNet: 60924:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.982825] LNetError: 60924:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.986645] LNetError: 60924:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 542.991284] LNet: Removed LNI 192.168.204.13@tcp [ 543.000648] LNet: Removed LNI 192.168.204.13@tcp1 [ 543.354236] Key type .llcrypt unregistered [ 543.356085] Key type ._llcrypt unregistered [ 545.764150] Key type ._llcrypt registered [ 545.766056] Key type .llcrypt registered [ 545.867305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 545.876598] alg: No test for adler32 (adler32-zlib) [ 546.683315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 546.697062] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 551.028121] Lustre: DEBUG MARKER: Initial discovery [ 551.075978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.113@tcp [ 552.176146] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 552.194372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 552.207572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.113@tcp [ 552.215495] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 552.222917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 552.237630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.113@tcp -r 1 -m GET -l 3 [ 552.250653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.113@tcp -r 1 -m GET -e local_error [ 552.262845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.113@tcp -r 1 -m PUT -l 6 [ 552.851615] Lustre: DEBUG MARKER: Force oleg413-client.virtnet to discover 192.168.204.113@tcp (in background) [ 552.888683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.113@tcp [ 553.511105] Lustre: DEBUG MARKER: Force oleg413-server to discover 192.168.204.13@tcp [ 554.622310] Lustre: DEBUG MARKER: Wait for 61674 [ 555.935392] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 556.565752] Lustre: DEBUG MARKER: Finished wait on 61674 [ 556.614815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 556.641068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 556.771772] LNet: 62128:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.774813] LNetError: 62128:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.782523] LNet: Removed LNI 192.168.204.13@tcp [ 556.968478] Key type .llcrypt unregistered [ 556.969799] Key type ._llcrypt unregistered [ 563.202799] Key type ._llcrypt registered [ 563.203836] Key type .llcrypt registered [ 563.916586] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:25:49 (1756311949) [ 564.156406] Key type .llcrypt unregistered [ 564.157587] Key type ._llcrypt unregistered [ 566.395410] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 566.453386] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 566.454905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 566.456537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 568.777508] Key type ._llcrypt registered [ 568.779788] Key type .llcrypt registered [ 568.883469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 568.891313] alg: No test for adler32 (adler32-zlib) [ 569.718624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.747269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 569.759614] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 570.157283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 570.170653] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 574.576268] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:26:00 (1756311960) [ 574.637975] LNet: 64184:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.642695] LNetError: 64184:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.646931] LNet: Removed LNI 192.168.204.13@tcp [ 574.655804] LNet: Removed LNI 10.1.2.3@tcp [ 575.021599] Key type .llcrypt unregistered [ 575.023428] Key type ._llcrypt unregistered [ 577.551294] Key type ._llcrypt registered [ 577.552114] Key type .llcrypt registered [ 577.611779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.616209] alg: No test for adler32 (adler32-zlib) [ 578.454865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.489353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 578.504756] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 578.856874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 578.872547] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 579.374853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.13@tcp [ 580.447242] LNetError: Unexpected error -22 connecting to 192.168.204.13@tcp at host 192.168.204.13:988 [ 582.264600] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:26:08 (1756311968) [ 582.310917] LNet: 65310:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.313586] LNetError: 65310:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.315447] LNet: Removed LNI 192.168.204.13@tcp [ 582.324614] LNet: Removed LNI 10.1.2.3@tcp [ 582.513538] Key type .llcrypt unregistered [ 582.514665] Key type ._llcrypt unregistered [ 584.486266] Key type ._llcrypt registered [ 584.488232] Key type .llcrypt registered [ 584.558291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.566859] alg: No test for adler32 (adler32-zlib) [ 585.398407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.424809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 585.437598] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 585.836604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 585.850863] LNet: Added LNI 192.168.204.13@tcp2 [8/256/0/180] [ 586.365697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.13@tcp1 --nid 192.168.204.13@tcp2 [ 588.134711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp1 192.168.204.13@tcp2 [ 588.147388] LNetError: 66134:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.13@tcp1 to 192.168.204.13@tcp2 - Route table may be misconfigured [ 588.161469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp2 192.168.204.13@tcp1 [ 591.268399] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:26:17 (1756311977) [ 591.332519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.346414] LNet: 66703:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.349624] LNetError: 66703:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.352287] LNet: Removed LNI 192.168.204.13@tcp1 [ 592.422340] LNet: Removed LNI 192.168.204.13@tcp2 [ 592.435927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.468099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.483149] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 592.857965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 592.868502] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 593.322374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 593.345233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.13@tcp [ 593.365082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp1 192.168.204.13@tcp1 [ 593.386345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.13@tcp -d 192.168.204.13@tcp -e network_timeout [ 593.406941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -e network_timeout [ 593.430178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -e network_timeout [ 593.452302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -e network_timeout [ 593.472107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 593.481969] LNet: There was an unexpected network error while writing to 192.168.204.13: rc = -22 [ 593.492208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 593.503135] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 593.539919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 593.588513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 596.406951] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:26:22 (1756311982) [ 596.472479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.486828] LNet: 67560:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.490568] LNetError: 67560:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.494058] LNetError: 67560:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 596.497539] LNet: Removed LNI 192.168.204.13@tcp [ 597.541863] LNet: Removed LNI 192.168.204.13@tcp1 [ 597.555254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.601337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 597.742964] LNet: 67598:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.975212] Key type .llcrypt unregistered [ 597.977231] Key type ._llcrypt unregistered [ 601.964168] Key type ._llcrypt registered [ 601.965273] Key type .llcrypt registered [ 602.563048] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:26:28 (1756311988) [ 602.760768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.768045] alg: No test for adler32 (adler32-zlib) [ 603.568726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.141418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 604.962755] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:26:31 (1756311991) [ 605.019407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.034515] LNet: 68688:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.054476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.087475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 605.100968] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 605.477996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 605.490444] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 605.953273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 605.973890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp1 [ 605.994193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp1 [ 608.902702] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:26:35 (1756311995) [ 609.738360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg413-server [ 610.534437] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:26:36 (1756311996) [ 611.416450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg413-server [ 612.226550] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:26:38 (1756311998) [ 613.103521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg413-server [ 613.906475] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:26:40 (1756312000) [ 614.808065] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg413-server [ 615.627247] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:26:41 (1756312001) [ 616.512545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg413-server [ 617.344174] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:26:43 (1756312003) [ 618.233739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg413-server [ 619.050631] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:26:45 (1756312005) [ 619.947429] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg413-server [ 620.738837] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:26:46 (1756312006) [ 621.600903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg413-server [ 622.416837] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:26:48 (1756312008) [ 623.294699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg413-server [ 624.110304] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:26:50 (1756312010) [ 624.177587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.192479] LNet: 72556:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.196473] LNetError: 72556:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.200240] LNet: Removed LNI 192.168.204.13@tcp [ 625.253806] LNet: Removed LNI 192.168.204.13@tcp1 [ 625.265854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.296876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.311222] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 625.696567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 625.742401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 625.836916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.849716] LNet: 72702:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.853913] LNetError: 72702:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.857842] LNetError: 72702:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 625.867565] LNet: Removed LNI 192.168.204.13@tcp [ 625.878836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.909870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 625.924740] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 626.323815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 626.418481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.493538] LNet: Removed LNI 192.168.204.13@tcp [ 627.504890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.536050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.550925] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 627.931710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 627.978974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 629.147168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.159653] LNet: 72998:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.162996] LNet: 72998:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 629.167744] LNetError: 72998:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.171740] LNetError: 72998:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 629.185551] LNet: Removed LNI 192.168.204.13@tcp [ 629.196920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.227571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 629.242387] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 629.650559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 629.757991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.821564] LNet: Removed LNI 192.168.204.13@tcp [ 630.832962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.864229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 630.879531] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 631.258524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 631.309496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 631.423420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.436312] LNetError: 73284:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 631.440202] LNetError: 73284:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 631.453753] LNet: Removed LNI 192.168.204.13@tcp [ 631.466370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.498098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 631.514096] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 631.952609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 632.072609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.125669] LNet: Removed LNI 192.168.204.13@tcp [ 633.138133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.171648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 633.187737] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 633.597154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 633.646628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 633.771524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.784693] LNet: 73570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.788481] LNet: 73570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 634.853572] LNet: Removed LNI 192.168.204.13@tcp [ 634.865724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.896596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 634.912529] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 635.336781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 635.465851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.479162] LNetError: 73713:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.483246] LNetError: 73713:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 635.491690] LNet: Removed LNI 192.168.204.13@tcp [ 635.503313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.534953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.553623] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 635.939905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 635.991697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 637.199389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.245758] LNet: Removed LNI 192.168.204.13@tcp [ 638.257470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.290855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 638.307855] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 638.754863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 639.973787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.997755] LNet: Removed LNI 192.168.204.13@tcp [ 641.009973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.042928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.057289] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 641.456646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 641.499828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 641.628491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.661719] LNet: Removed LNI 192.168.204.13@tcp [ 642.675407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.709125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 642.726595] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 643.172759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 643.322044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.335330] LNet: 74305:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.339152] LNet: 74305:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 644.389612] LNet: Removed LNI 192.168.204.13@tcp [ 644.401982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.434102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.449607] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 644.870922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 644.910650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.924100] LNetError: 74435:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.928447] LNetError: 74435:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 644.940633] LNet: Removed LNI 192.168.204.13@tcp [ 644.951904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.982729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 644.996440] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 645.420747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 648.680934] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:27:14 (1756312034) [ 648.737367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.754737] LNet: Removed LNI 192.168.204.13@tcp [ 648.767375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.793258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 648.804330] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 649.330339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.356871] LNet: Removed LNI 192.168.204.13@tcp231 [ 649.368459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.406591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 649.419639] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 649.422440] LNet: 75279:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 649.549760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 649.563649] LNet: Removed LNI 192.168.204.13@tcp231 [ 649.569927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 649.579070] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 650.076551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.093294] LNet: Removed LNI 192.168.204.13@tcp231 [ 650.101574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.132544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 650.146078] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/0] [ 650.148579] LNet: 75429:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 650.280290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 650.294633] LNet: Removed LNI 192.168.204.13@tcp231 [ 650.301817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 650.312467] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/0] [ 650.815388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.829625] LNet: Removed LNI 192.168.204.13@tcp231 [ 650.837196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.857292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 650.864548] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/60] [ 650.971879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 650.989392] LNet: Removed LNI 192.168.204.13@tcp231 [ 650.996584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 651.009394] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/60] [ 651.516892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.539626] LNet: Removed LNI 192.168.204.13@tcp231 [ 651.551245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.590464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 651.604936] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 651.608475] LNet: 75725:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 651.612924] LNet: 75725:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 651.741817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 651.758572] LNet: Removed LNI 192.168.204.13@tcp231 [ 651.763108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 651.769971] LNet: Added LNI 192.168.204.13@tcp231 [8/256/0/180] [ 654.969578] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:27:21 (1756312041) [ 655.041115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.067984] LNet: Removed LNI 192.168.204.13@tcp231 [ 655.079866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.116773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 655.130400] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 655.493088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 655.514239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 655.538820] LNet: Removed LNI 192.168.204.13@tcp232 [ 655.548583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 655.563515] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 655.952574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 655.975673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 655.999611] LNet: Removed LNI 192.168.204.13@tcp232 [ 656.009552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 656.024409] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 656.406217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 656.427945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 656.449271] LNet: Removed LNI 192.168.204.13@tcp232 [ 656.459536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 656.475378] LNet: Added LNI 192.168.204.13@tcp232 [8/256/0/180] [ 656.822690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 656.844823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 656.866854] LNet: Removed LNI 192.168.204.13@tcp232 [ 656.875602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 659.787843] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:27:25 (1756312045) [ 659.994288] LNet: 77546:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.998075] LNet: 77546:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 660.336134] Key type .llcrypt unregistered [ 660.337658] Key type ._llcrypt unregistered [ 662.633513] Key type ._llcrypt registered [ 662.634341] Key type .llcrypt registered [ 662.683210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 662.687318] alg: No test for adler32 (adler32-zlib) [ 663.496178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 663.508436] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 667.188910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 671.011676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.030559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 671.043631] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 671.797363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_interrupt -r 2 [ 671.815923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 2 [ 671.835165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_interrupt -r 2 [ 671.856945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 2 [ 671.878967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_interrupt -r 2 [ 671.901517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_interrupt -r 2 [ 671.921351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_interrupt -r 2 [ 671.942495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_interrupt -r 2 [ 671.960493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 671.971500] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 671.975275] LNet: 1 local NIs in recovery (showing 1): 192.168.204.13@tcp [ 672.131149] LNet: 78191:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 672.166258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 672.185377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 672.471893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_dropped -r 2 [ 672.493478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 2 [ 672.514327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_dropped -r 2 [ 672.534525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 2 [ 672.555332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_dropped -r 2 [ 672.574876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_dropped -r 2 [ 672.595855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_dropped -r 2 [ 672.614686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_dropped -r 2 [ 672.634599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 672.648152] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 672.821732] LNet: 78299:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 672.857368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 672.877939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 673.158525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_aborted -r 2 [ 673.176405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 2 [ 673.194780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_aborted -r 2 [ 673.211534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 2 [ 673.227562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_aborted -r 2 [ 673.240964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_aborted -r 2 [ 673.256803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_aborted -r 2 [ 673.270539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_aborted -r 2 [ 673.286300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 673.462032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 673.477627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 673.728460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_no_route -r 2 [ 673.747214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 2 [ 673.766580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_no_route -r 2 [ 673.784737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 2 [ 673.801495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_no_route -r 2 [ 673.816367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_no_route -r 2 [ 673.829755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_no_route -r 2 [ 673.841719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_no_route -r 2 [ 673.853205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 673.860197] LNet: There was an unexpected network error while writing to 192.168.204.113: rc = -22 [ 673.863085] LNet: Skipped 2 previous similar messages [ 673.987922] LNet: 78516:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 673.990850] LNet: 78516:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 674.023562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 674.041686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 674.323318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e local_timeout -r 2 [ 674.344193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 2 [ 674.367205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e local_timeout -r 2 [ 674.388875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 2 [ 674.409368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e local_timeout -r 2 [ 674.427350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e local_timeout -r 2 [ 674.443798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e local_timeout -r 2 [ 674.458913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e local_timeout -r 2 [ 674.472024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 674.622453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 674.635678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 674.875207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp -m GET -e remote_dropped -r 2 [ 674.894439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 2 [ 674.913870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp -m GET -e remote_dropped -r 2 [ 674.932359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.13@tcp -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 2 [ 674.949963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp -m GET -e remote_dropped -r 2 [ 674.965705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.113@tcp1 -m GET -e remote_dropped -r 2 [ 674.979918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp -m GET -e remote_dropped -r 2 [ 674.992870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.13@tcp1 -d 192.168.204.13@tcp1 -m GET -e remote_dropped -r 2 [ 675.007096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.13@tcp 192.168.204.113@tcp [ 675.017425] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 675.163473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 675.176472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 677.764346] LNet: 78800:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.768876] LNetError: 78800:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.773936] LNet: Removed LNI 192.168.204.13@tcp [ 677.784583] LNet: Removed LNI 192.168.204.13@tcp1 [ 678.021917] Key type .llcrypt unregistered [ 678.024057] Key type ._llcrypt unregistered [ 684.595596] Key type ._llcrypt registered [ 684.597222] Key type .llcrypt registered [ 685.296227] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:27:51 (1756312071) [ 685.478120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.483591] alg: No test for adler32 (adler32-zlib) [ 686.298704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.326699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.340674] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 686.755241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 686.796449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 686.826622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 686.872340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 686.907487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 686.933409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 689.796495] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:27:55 (1756312075) [ 689.848865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.864241] LNet: 80350:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.868631] LNetError: 80350:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.877634] LNet: Removed LNI 192.168.204.13@tcp [ 689.894990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.926307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.940499] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 693.134204] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:27:59 (1756312079) [ 693.846402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 694.647129] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:28:00 (1756312080) [ 694.864808] LNet: 81386:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.868700] LNetError: 81386:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.876801] LNet: Removed LNI 192.168.204.13@tcp [ 695.229584] Key type .llcrypt unregistered [ 695.231464] Key type ._llcrypt unregistered [ 697.235072] Key type ._llcrypt registered [ 697.236915] Key type .llcrypt registered [ 697.333242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.339467] alg: No test for adler32 (adler32-zlib) [ 698.141235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 698.153625] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 701.920597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 703.855580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.113@tcp [ 703.934705] LNet: 81797:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.936471] LNetError: 81797:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.945261] LNet: Removed LNI 192.168.204.13@tcp [ 704.085249] Key type .llcrypt unregistered [ 704.086125] Key type ._llcrypt unregistered [ 708.398949] Key type ._llcrypt registered [ 708.399750] Key type .llcrypt registered [ 708.989503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 709.763691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 710.540406] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:28:16 (1756312096) [ 710.720890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.727840] alg: No test for adler32 (adler32-zlib) [ 711.536825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.575478] LNet: 82879:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.878353] Key type .llcrypt unregistered [ 711.880248] Key type ._llcrypt unregistered [ 714.171267] Key type ._llcrypt registered [ 714.172147] Key type .llcrypt registered [ 714.216509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.220399] alg: No test for adler32 (adler32-zlib) [ 715.032368] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 715.034164] LNet: 83131:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 715.271793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.439132] LNetError: 83143:0:(lib-lnet.h:1309:lnet_set_route_aliveness()) route to o2ib through 192.168.204.14@tcp has gone from up to down [ 721.444896] LNet: 83189:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.449227] LNetError: 83189:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.459579] LNet: Removed LNI 192.168.204.13@tcp [ 721.473330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.505827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 721.520094] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 722.209459] Lustre: DEBUG MARKER: sanity-lnet test_255: @@@@@@ FAIL: Import failed 234 [ 726.943597] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:28:33 (1756312113) [ 727.838449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg413-server [ 728.652414] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:28:34 (1756312114) [ 729.532069] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg413-server [ 730.341856] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:28:36 (1756312116) [ 730.949213] Lustre: Lustre: Build Version: 2.16.57_134_ga204b13 [ 732.607126] Key type lgssc registered [ 733.143239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 743.486334] Key type lgssc unregistered [ 743.704127] LNet: 85729:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.708636] LNetError: 85729:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.718738] LNet: Removed LNI 192.168.204.13@tcp [ 744.074569] Key type .llcrypt unregistered [ 744.076820] Key type ._llcrypt unregistered [ 750.809929] Key type ._llcrypt registered [ 750.810860] Key type .llcrypt registered [ 751.373155] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:28:57 (1756312137) [ 751.519259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.524932] alg: No test for adler32 (adler32-zlib) [ 752.403310] LNetError: 86559:0:(api-ni.c:2623:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 752.612700] Key type .llcrypt unregistered [ 752.614628] Key type ._llcrypt unregistered [ 754.960511] Key type ._llcrypt registered [ 754.961546] Key type .llcrypt registered [ 755.503080] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:29:01 (1756312141) [ 755.732658] Key type .llcrypt unregistered [ 755.734403] Key type ._llcrypt unregistered [ 758.026311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.027586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.234956] Key type ._llcrypt registered [ 760.236582] Key type .llcrypt registered [ 760.322325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 760.328795] alg: No test for adler32 (adler32-zlib) [ 761.136359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.157898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 766.049949] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:29:12 (1756312152) [ 766.089612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.100373] LNet: 88322:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.103276] LNetError: 88322:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 766.109711] LNet: Removed LNI 10.1.2.3@tcp [ 766.122041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.471955] Key type .llcrypt unregistered [ 766.474185] Key type ._llcrypt unregistered [ 768.812320] Key type ._llcrypt registered [ 768.813180] Key type .llcrypt registered [ 768.857864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.862549] alg: No test for adler32 (adler32-zlib) [ 769.668288] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 769.696659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 769.771252] LNet: 88611:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.775567] LNetError: 88611:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.784681] LNet: Removed LNI 192.168.204.13@tcp [ 770.128575] Key type .llcrypt unregistered [ 770.130155] Key type ._llcrypt unregistered [ 774.669088] Key type ._llcrypt registered [ 774.669982] Key type .llcrypt registered [ 775.197582] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:29:21 (1756312161) [ 775.443257] Key type .llcrypt unregistered [ 775.445370] Key type ._llcrypt unregistered [ 777.581918] Key type ._llcrypt registered [ 777.582741] Key type .llcrypt registered [ 777.633381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.637411] alg: No test for adler32 (adler32-zlib) [ 781.325909] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:29:27 (1756312167) [ 781.398345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.423521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.453377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.467556] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 781.807807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.827795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 781.849603] LNet: Removed LNI 192.168.204.13@tcp [ 781.990820] LNet: 90341:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.212587] Key type .llcrypt unregistered [ 782.214622] Key type ._llcrypt unregistered [ 786.649450] Key type ._llcrypt registered [ 786.651259] Key type .llcrypt registered [ 787.307701] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:29:33 (1756312173) [ 787.487221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 787.492969] alg: No test for adler32 (adler32-zlib) [ 788.296313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.322658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.337882] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 788.723875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 788.743263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.13@tcp [ 788.753111] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 788.755705] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 788.757951] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 788, 0, 0 [ 788.760978] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 788.762676] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 788.764923] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 788.766787] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 788, 0, 0 [ 788.769350] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 788.770737] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 788.772618] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 788.774266] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 788, 0, 838 [ 788.776552] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 788.777821] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 788.779493] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 788.780919] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 788, 0, 838 [ 788.783009] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 791.678406] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:29:37 (1756312177) [ 791.899211] LNet: 91872:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.903438] LNetError: 91872:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.915304] LNet: Removed LNI 192.168.204.13@tcp [ 792.251521] Key type .llcrypt unregistered [ 792.252871] Key type ._llcrypt unregistered [ 794.731106] Key type ._llcrypt registered [ 794.731910] Key type .llcrypt registered [ 794.785662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.789958] alg: No test for adler32 (adler32-zlib) [ 795.603413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 795.620637] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 799.292593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 802.846412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.867710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 802.882430] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 803.423321] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.492171] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.493750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.494994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.681193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 805.694144] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 811.002394] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:29:57 (1756312197) [ 811.068774] LNet: 93360:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.072735] LNetError: 93360:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.076708] LNet: Removed LNI 192.168.204.13@tcp [ 812.127251] LNet: Removed LNI 192.168.204.13@tcp1 [ 812.129482] LNetError: 93360:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 812.133577] LNetError: 93360:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 812.145728] LNet: Removed LNI 10.1.2.3@tcp99 [ 812.537703] Key type .llcrypt unregistered [ 812.539656] Key type ._llcrypt unregistered [ 815.095495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.096859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.323297] Key type ._llcrypt registered [ 817.325166] Key type .llcrypt registered [ 817.428786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.435577] alg: No test for adler32 (adler32-zlib) [ 818.272699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.305350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.319590] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 818.739076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 818.751982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.983215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 820.012276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 820.038229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 820.051538] LNetError: 94099:0:(peer.c:2202:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 820.072489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 820.102770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 820.257810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 820.302833] LNet: 94119:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.308481] LNetError: 94119:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.312770] LNet: Removed LNI 192.168.204.13@tcp [ 820.322818] LNet: Removed LNI 10.1.2.3@tcp [ 820.694596] Key type .llcrypt unregistered [ 820.696527] Key type ._llcrypt unregistered [ 825.266954] Key type ._llcrypt registered [ 825.268098] Key type .llcrypt registered [ 825.896513] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:30:12 (1756312212) [ 826.093780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.100964] alg: No test for adler32 (adler32-zlib) [ 826.923456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.950419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.965310] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 827.350166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.13@tcp [ 827.393991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg413-client@tcp [ 830.502825] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:30:16 (1756312216) [ 830.573374] LNet: 95639:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.578334] LNetError: 95639:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.598091] LNet: Removed LNI 192.168.204.13@tcp [ 830.966569] Key type .llcrypt unregistered [ 830.968532] Key type ._llcrypt unregistered [ 833.565429] Key type ._llcrypt registered [ 833.566256] Key type .llcrypt registered [ 833.612625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.617271] alg: No test for adler32 (adler32-zlib) [ 834.444067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 834.459442] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 834.481982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 836.769822] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:30:23 (1756312223) [ 837.207936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 837.698405] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:30:24 (1756312224) [ 838.171179] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 838.741123] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:30:25 (1756312225) [ 838.772555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.781474] LNet: 97052:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.783341] LNetError: 97052:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.789847] LNet: Removed LNI 192.168.204.13@tcp [ 838.800395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.822827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.833236] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 839.086984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 839.102936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 841.538759] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:30:27 (1756312227) [ 841.567697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.577514] LNet: 97756:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.579813] LNetError: 97756:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.586599] LNet: Removed LNI 192.168.204.13@tcp [ 841.595946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.626198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 841.644051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 841.948287] Key type .llcrypt unregistered [ 841.949852] Key type ._llcrypt unregistered [ 846.865800] Key type ._llcrypt registered [ 846.867270] Key type .llcrypt registered [ 847.467919] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:30:33 (1756312233) [ 847.632633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.639711] alg: No test for adler32 (adler32-zlib) [ 848.454204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.476878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.490952] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 848.831336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 848.863708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 851.669591] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:30:37 (1756312237) [ 851.732193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.747293] LNet: 99308:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.750818] LNetError: 99308:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.772718] LNet: Removed LNI 192.168.204.13@tcp [ 851.783139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.806922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 851.821673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 854.408427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 855.022798] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:30:41 (1756312241) [ 855.065541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.076819] LNet: 100023:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.090466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.215584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.228991] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 855.533180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 855.543444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 855.966284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.13@tcp [ 855.984400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 856.012443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.13@tcp -m PUT -e local_timeout -r 1 [ 856.048347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 856.077775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 856.079918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 856.180216] LNetError: 100060:0:(lib-move.c:4945:lnet_parse()) 192.168.204.13@tcp, src 192.168.204.13@tcp: Dropping PUT (error -108 looking up sender) [ 857.183545] LNet: 100335:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.186521] LNetError: 100335:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.189138] LNet: Removed LNI 192.168.204.13@tcp [ 857.196618] LNet: Removed LNI 10.1.2.3@tcp [ 857.439735] Key type .llcrypt unregistered [ 857.441048] Key type ._llcrypt unregistered [ 862.047951] Key type ._llcrypt registered [ 862.049042] Key type .llcrypt registered [ 862.653104] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:30:48 (1756312248) [ 862.818826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.827844] alg: No test for adler32 (adler32-zlib) [ 863.668765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.888884] LNet: 101183:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.212392] Key type .llcrypt unregistered [ 864.213656] Key type ._llcrypt unregistered [ 866.803066] Key type ._llcrypt registered [ 866.804722] Key type .llcrypt registered [ 866.902484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.913306] alg: No test for adler32 (adler32-zlib) [ 867.748972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 867.764575] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 871.183664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 871.277078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 871.576474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.113@tcp [ 871.754731] LNet: 101619:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 871.796115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 871.815729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 872.153193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 872.317123] LNet: 101724:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 872.351803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 872.369666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 872.690812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.113@tcp [ 872.924238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 872.927159] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.113@tcp [ 872.941820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 873.282538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 873.467263] LNet: 101934:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 873.469925] LNet: 101934:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 873.515730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 873.536971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 873.702945] LNet: 101975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.706429] LNetError: 101975:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.719596] LNet: Removed LNI 192.168.204.13@tcp [ 873.936961] Key type .llcrypt unregistered [ 873.938839] Key type ._llcrypt unregistered [ 880.661832] Key type ._llcrypt registered [ 880.663602] Key type .llcrypt registered [ 881.279135] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:31:07 (1756312267) [ 881.455897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.466618] alg: No test for adler32 (adler32-zlib) [ 882.292671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.486876] LNet: 102864:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.795521] Key type .llcrypt unregistered [ 882.797116] Key type ._llcrypt unregistered [ 885.163320] Key type ._llcrypt registered [ 885.164383] Key type .llcrypt registered [ 885.248324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 885.255676] alg: No test for adler32 (adler32-zlib) [ 886.085246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 886.099865] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 889.207441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.113@tcp [ 892.080729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.095770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 892.105784] LNet: Added LNI 192.168.204.13@tcp1 [8/256/0/180] [ 892.486617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 892.738584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.113@tcp [ 892.900631] LNet: 103498:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 892.939237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 892.954744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 893.240719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 893.279215] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.113@tcp, 192.168.204.113@tcp1 [ 893.401976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 893.419129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 893.681203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.113@tcp [ 893.831491] LNet: 103708:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 893.834813] LNet: 103708:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 893.869938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 893.885261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 894.153466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 894.330472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 894.343987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 896.578852] LNet: 103889:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.581444] LNetError: 103889:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.584335] LNet: Removed LNI 192.168.204.13@tcp [ 896.589325] LNet: Removed LNI 192.168.204.13@tcp1 [ 896.748895] Key type .llcrypt unregistered [ 896.750465] Key type ._llcrypt unregistered [ 902.326827] Key type ._llcrypt registered [ 902.328180] Key type .llcrypt registered [ 902.762896] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 800 sec =========== 12:31:29 (1756312289) [ 902.984467] Key type .llcrypt unregistered [ 902.985920] Key type ._llcrypt unregistered [ 906.944338] Key type ._llcrypt registered [ 906.945530] Key type .llcrypt registered [ 907.137735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 907.143546] alg: No test for adler32 (adler32-zlib) [ 908.011577] Lustre: Lustre: Build Version: 2.16.57_134_ga204b13 [ 908.110922] LNet: Added LNI 192.168.204.13@tcp [8/256/0/180] [ 909.703202] Key type lgssc registered [ 910.102929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 932.585537] Lustre: Mounted lustre-client [ 935.163283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 942.405489] LustreError: 107103:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff97d189738000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 942.411530] LustreError: 107103:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 942.424135] Lustre: Unmounted lustre-client [ 954.075180] Key type lgssc unregistered [ 954.195778] LNet: 107584:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.200194] LNetError: 107584:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.209261] LNet: Removed LNI 192.168.204.13@tcp [ 954.484102] Key type .llcrypt unregistered [ 954.484889] Key type ._llcrypt unregistered