[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 433267481 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002338] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009120] LSM: Security Framework initializing [ 0.010027] Yama: becoming mindful. [ 0.011025] SELinux: Initializing. [ 0.012057] *** VALIDATE selinux *** [ 0.021116] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026951] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028205] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029068] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030067] *** VALIDATE tmpfs *** [ 0.031297] *** VALIDATE proc *** [ 0.032175] *** VALIDATE cgroup *** [ 0.033004] *** VALIDATE cgroup2 *** [ 0.035072] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038020] Spectre V2 : User space: Vulnerable [ 0.039004] Speculative Store Bypass: Vulnerable [ 0.042052] debug: unmapping init [mem 0xffffffffb2e59000-0xffffffffb2e60fff] [ 0.044794] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046017] ... version: 2 [ 0.046949] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048006] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.052986] rcu: Hierarchical SRCU implementation. [ 0.054270] smp: Bringing up secondary CPUs ... [ 0.055499] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.060111] smp: Brought up 1 node, 4 CPUs [ 0.062026] smpboot: Max logical packages: 1 [ 0.063007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.253022] node 0 deferred pages initialised in 189ms [ 0.256124] devtmpfs: initialized [ 0.257243] x86/mm: Memory block size: 128MB [ 0.260605] gcov: version magic: 0x41383552 [ 0.262375] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.263061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.264264] pinctrl core: initialized pinctrl subsystem [ 0.265295] [ 0.265856] ************************************************************* [ 0.267014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269015] ** ** [ 0.271014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.273016] ** ** [ 0.275012] ** This means that this kernel is built to expose internal ** [ 0.277014] ** IOMMU data structures, which may compromise security on ** [ 0.279023] ** your system. ** [ 0.282021] ** ** [ 0.283009] ** If you see this message and you are not debugging the ** [ 0.286013] ** kernel, report this immediately to your vendor! ** [ 0.287009] ** ** [ 0.289008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290015] ************************************************************* [ 0.292932] NET: Registered protocol family 16 [ 0.294440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.295052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.298135] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.301162] cpuidle: using governor menu [ 0.302517] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.304502] PCI: Using configuration type 1 for base access [ 0.305152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312046] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.316225] cryptd: max_cpu_qlen set to 1000 [ 0.318200] ACPI: Added _OSI(Module Device) [ 0.319080] ACPI: Added _OSI(Processor Device) [ 0.321011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.322010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326426] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.331363] ACPI: Interpreter enabled [ 0.332044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333009] ACPI: Using IOAPIC for interrupt routing [ 0.333985] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.336295] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.343926] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350041] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352840] acpiphp: Slot [2] registered [ 0.354160] acpiphp: Slot [3] registered [ 0.355078] acpiphp: Slot [4] registered [ 0.357077] acpiphp: Slot [5] registered [ 0.358110] acpiphp: Slot [6] registered [ 0.360085] acpiphp: Slot [7] registered [ 0.361082] acpiphp: Slot [8] registered [ 0.362065] acpiphp: Slot [9] registered [ 0.364063] acpiphp: Slot [10] registered [ 0.365058] acpiphp: Slot [11] registered [ 0.366064] acpiphp: Slot [12] registered [ 0.367064] acpiphp: Slot [13] registered [ 0.368000] acpiphp: Slot [14] registered [ 0.370070] acpiphp: Slot [15] registered [ 0.372062] acpiphp: Slot [16] registered [ 0.372942] acpiphp: Slot [17] registered [ 0.374053] acpiphp: Slot [18] registered [ 0.374927] acpiphp: Slot [19] registered [ 0.375094] acpiphp: Slot [20] registered [ 0.377104] acpiphp: Slot [21] registered [ 0.378081] acpiphp: Slot [22] registered [ 0.379113] acpiphp: Slot [23] registered [ 0.380092] acpiphp: Slot [24] registered [ 0.382079] acpiphp: Slot [25] registered [ 0.383064] acpiphp: Slot [26] registered [ 0.384081] acpiphp: Slot [27] registered [ 0.385065] acpiphp: Slot [28] registered [ 0.386090] acpiphp: Slot [29] registered [ 0.387064] acpiphp: Slot [30] registered [ 0.388062] acpiphp: Slot [31] registered [ 0.390071] PCI host bridge to bus 0000:00 [ 0.391019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.395016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.397018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.400031] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.402026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.404183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.408395] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412490] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.422054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.424016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.426020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.428016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.430383] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.432753] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.435034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.437610] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.441018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.448559] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.452015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.455070] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.461025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.469020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481030] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.488264] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518021] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.527944] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.530277] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.532254] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.533317] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.535160] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.538160] iommu: Default domain type: Passthrough [ 0.539360] SCSI subsystem initialized [ 0.540115] ACPI: bus type USB registered [ 0.541085] usbcore: registered new interface driver usbfs [ 0.543061] usbcore: registered new interface driver hub [ 0.544064] usbcore: registered new device driver usb [ 0.545124] pps_core: LinuxPPS API ver. 1 registered [ 0.547012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.549089] PTP clock support registered [ 0.551183] EDAC MC: Ver: 3.0.0 [ 0.552525] PCI: Using ACPI for IRQ routing [ 0.554448] NetLabel: Initializing [ 0.556022] NetLabel: domain hash size = 128 [ 0.557008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.558097] NetLabel: unlabeled traffic allowed by default [ 0.561145] vgaarb: loaded [ 0.562263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.564017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.567568] clocksource: Switched to clocksource kvm-clock [ 0.662367] VFS: Disk quotas dquot_6.6.0 [ 0.663899] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.666184] *** VALIDATE ramfs *** [ 0.667326] *** VALIDATE hugetlbfs *** [ 0.668544] pnp: PnP ACPI init [ 0.670962] pnp: PnP ACPI: found 6 devices [ 0.684852] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.687585] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.689262] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.690925] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.692840] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.694366] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.696252] NET: Registered protocol family 2 [ 0.697767] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.701917] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.707983] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.714965] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.721719] TCP: Hash tables configured (established 65536 bind 65536) [ 0.726271] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.729426] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.732188] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.735985] NET: Registered protocol family 1 [ 0.738502] RPC: Registered named UNIX socket transport module. [ 0.740691] RPC: Registered udp transport module. [ 0.741940] RPC: Registered tcp transport module. [ 0.743456] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.745546] NET: Registered protocol family 44 [ 0.747630] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.749356] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.751249] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.753181] PCI: CLS 0 bytes, default 64 [ 0.754703] Unpacking initramfs... [ 2.189146] debug: unmapping init [mem 0xffff991afcc64000-0xffff991afffcffff] [ 2.192642] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.194810] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.197234] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.702930] Initialise system trusted keyrings [ 2.704445] Key type blacklist registered [ 2.706350] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.716034] zbud: loaded [ 2.719056] *** VALIDATE nfs *** [ 2.720166] *** VALIDATE nfs4 *** [ 2.721688] pstore: using deflate compression [ 2.725132] Platform Keyring initialized [ 2.855621] NET: Registered protocol family 38 [ 2.858022] Key type asymmetric registered [ 2.859315] Asymmetric key parser 'x509' registered [ 2.861456] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.864819] io scheduler mq-deadline registered [ 2.867646] io scheduler kyber registered [ 2.869345] io scheduler bfq registered [ 2.874837] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.877642] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.883599] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.886614] ACPI: Power Button [PWRF] [ 2.995386] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.097750] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.213333] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.244727] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.275378] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.280243] Non-volatile memory driver v1.3 [ 3.281587] Linux agpgart interface v0.103 [ 3.317741] virtio_blk virtio1: [vda] 133040 512-byte logical blocks (68.1 MB/65.0 MiB) [ 3.320680] vda: detected capacity change from 0 to 68116480 [ 3.341870] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.347989] vdb: detected capacity change from 0 to 1073741824 [ 3.361374] libphy: Fixed MDIO Bus: probed [ 3.370618] usbcore: registered new interface driver usbserial_generic [ 3.372911] usbserial: USB Serial support registered for generic [ 3.374943] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.384042] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.388497] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.394814] mousedev: PS/2 mouse device common for all mice [ 3.398542] rtc_cmos 00:05: RTC can wake from S4 [ 3.400813] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.420973] rtc_cmos 00:05: registered as rtc0 [ 3.422477] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.425751] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.427757] intel_pstate: CPU model not supported [ 3.433019] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.437470] hid: raw HID events driver (C) Jiri Kosina [ 3.439218] usbcore: registered new interface driver usbhid [ 3.440959] usbhid: USB HID core driver [ 3.442142] drop_monitor: Initializing network drop monitor service [ 3.444189] Initializing XFRM netlink socket [ 3.445878] NET: Registered protocol family 10 [ 3.449378] Segment Routing with IPv6 [ 3.450656] NET: Registered protocol family 17 [ 3.453661] mpls_gso: MPLS GSO support [ 3.459062] RAS: Correctable Errors collector initialized. [ 3.461072] AVX version of gcm_enc/dec engaged. [ 3.462697] AES CTR mode by8 optimization enabled [ 3.577451] sched_clock: Marking stable (3577428500, 0)->(4436635362, -859206862) [ 3.582849] registered taskstats version 1 [ 3.585702] Loading compiled-in X.509 certificates [ 3.588775] zswap: loaded using pool lzo/zbud [ 3.615135] Key type big_key registered [ 3.638929] Key type encrypted registered [ 3.640398] ima: No TPM chip found, activating TPM-bypass! [ 3.642319] ima: Allocated hash algorithm: sha1 [ 3.644019] ima: No architecture policies found [ 3.645690] evm: Initialising EVM extended attributes: [ 3.647625] evm: security.selinux [ 3.648889] evm: security.ima [ 3.649897] evm: security.capability [ 3.651404] evm: HMAC attrs: 0x1 [ 3.653910] rtc_cmos 00:05: setting system clock to 2025-08-31 21:30:09 UTC (1756675809) [ 3.661728] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.664946] debug: unmapping init [mem 0xffffffffb2b82000-0xffffffffb2e58fff] [ 3.672115] Write protecting the kernel read-only data: 28672k [ 3.679735] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 3.684657] debug: unmapping init [mem 0xffffffffb1b14000-0xffffffffb1bfffff] [ 3.751118] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.759273] systemd[1]: Detected virtualization kvm. [ 3.761348] systemd[1]: Detected architecture x86-64. [ 3.762671] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.789338] systemd[1]: No hostname configured. [ 3.791163] systemd[1]: Set hostname to . [ 3.793240] random: systemd: uninitialized urandom read (16 bytes read) [ 3.795499] systemd[1]: Initializing machine ID from random generator. [ 3.854162] random: ln: uninitialized urandom read (6 bytes read) [ 3.991713] random: systemd: uninitialized urandom read (16 bytes read) [ 3.995486] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.002838] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.009384] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.880172] device-mapper: uevent: version 1.0.3 [ 4.882240] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.934844] random: fast init done [ 5.953694] virtio_net virtio0 ens2: renamed from eth0 [ 6.122196] scsi host0: ata_piix [ 6.173617] scsi host1: ata_piix [ 6.176259] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.179188] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.158570] random: crng init done [ 11.160232] random: 7 urandom warning(s) missed due to ratelimiting [ 11.935770] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.997216] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.185671] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.547614] SELinux: Disabled at runtime. [ 15.629181] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.638656] systemd[1]: Detected virtualization kvm. [ 15.641598] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.578277] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.583320] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.589594] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.594278] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.598281] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.608447] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.620388] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on ini[ 16.801137] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.489843] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.096820] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.166165] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.578685] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.615884] EDAC sbridge: Ver: 1.1.2 [ 20.670615] Key type dns_resolver registered [ 21.080356] NFS: Registering the id_resolver key type [ 21.082308] Key type id_resolver registered [ 21.083804] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg443-client login: [ 53.010240] libcfs: loading out-of-tree module taints kernel. [ 53.074091] Key type ._llcrypt registered [ 53.075149] Key type .llcrypt registered [ 53.302029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 53.308167] alg: No test for adler32 (adler32-zlib) [ 54.258169] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 54.522819] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 56.127128] Key type lgssc registered [ 56.683524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.015930] Lustre: Mounted lustre-client [ 103.312820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 117.872308] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing check_logdir /tmp/testlogs/ [ 119.281264] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing yml_node [ 120.686640] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 121.454276] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 122.207845] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 122.687431] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Aug 31 17:32:08 EDT 2025 [ 126.944161] Lustre: lustre-OST0000-osc-ffff991b507fb800: disconnect after 24s idle [ 127.232997] LustreError: 7920:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff991b507fb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 127.236602] LustreError: 7920:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 127.245488] Lustre: Unmounted lustre-client [ 138.636400] Key type lgssc unregistered [ 138.763314] LNet: 8366:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 138.766679] LNetError: 8366:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 138.777503] LNet: Removed LNI 192.168.204.43@tcp [ 139.075183] Key type .llcrypt unregistered [ 139.076330] Key type ._llcrypt unregistered [ 148.006398] Key type ._llcrypt registered [ 148.007936] Key type .llcrypt registered [ 148.079988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 148.087555] alg: No test for adler32 (adler32-zlib) [ 148.900344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 148.930762] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 148.941192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 150.465335] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 150.502918] LNet: 9230:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 150.507557] LNetError: 9230:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 150.517628] LNet: Removed LNI 192.168.204.43@tcp [ 150.848772] Key type .llcrypt unregistered [ 150.850567] Key type ._llcrypt unregistered [ 153.040507] Key type ._llcrypt registered [ 153.042016] Key type .llcrypt registered [ 153.540733] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:32:38 (1756675958) [ 153.646173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 153.653407] alg: No test for adler32 (adler32-zlib) [ 154.473781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 154.509975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 154.519576] LNet: 9643:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 154.529278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 154.555973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 154.565336] LNet: 9650:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 157.460418] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:32:42 (1756675962) [ 157.508900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 157.521768] LNet: 10223:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 157.534712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 157.570450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 160.098463] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:32:45 (1756675965) [ 160.141363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 160.151818] LNet: 10807:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 160.163360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 160.203312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 162.856085] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:32:48 (1756675968) [ 162.903594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 162.913468] LNet: 11389:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 162.924620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 162.971040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 165.478769] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:32:50 (1756675970) [ 165.506662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.524318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.559518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 165.727875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 168.263629] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:32:53 (1756675973) [ 168.294577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.305395] LNet: 12561:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 168.308091] LNet: 12561:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 168.318709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.356323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 168.514771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 168.648517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 171.132737] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:32:56 (1756675976) [ 171.171905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 171.190621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 171.227598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 173.669251] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:32:59 (1756675979) [ 173.698914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.717572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.748352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 173.763961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 173.914198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 173.932261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 174.104568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 174.120869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 174.262591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 174.277826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 174.428908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 174.447786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 174.607467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 174.624122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 174.760657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 174.772785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 174.899027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 174.912231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 175.037598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 175.050821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 175.196173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 175.211181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 175.355676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 175.371338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 175.517180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 175.537774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 175.689780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 175.706630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 178.356342] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:33:03 (1756675983) [ 178.394808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 178.404405] LNet: 14422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 178.407635] LNet: 14422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 178.420411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 178.461651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 178.483806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 181.033848] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:33:06 (1756675986) [ 181.069763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.097157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.142630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 181.161732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 183.761549] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:33:09 (1756675989) [ 183.796809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.817710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.857734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 183.879164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 186.644240] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:33:11 (1756675991) [ 186.686497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.718916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.781306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 186.803919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 189.377772] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:33:14 (1756675994) [ 189.424077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.442882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.475058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 189.493455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 192.217094] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:33:17 (1756675997) [ 192.245661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 192.265515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 192.305982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 192.330129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 194.985207] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:33:20 (1756676000) [ 195.032755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 195.043540] LNet: 17938:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 195.046285] LNet: 17938:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 195.061290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 195.099408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 195.267845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 197.744657] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:33:23 (1756676003) [ 197.784988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.802815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.880422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 197.963111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 200.424896] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:33:25 (1756676005) [ 200.466152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.496866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.576267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 200.684654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 203.399799] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:33:28 (1756676008) [ 203.434338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.452711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.645416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 203.768347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 206.526904] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:33:31 (1756676011) [ 206.569829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.599714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.647625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 206.671755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 209.163895] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:33:34 (1756676014) [ 209.207653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.224153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.258046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 209.284115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 212.014300] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:33:37 (1756676017) [ 212.055661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.075642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 212.119466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 214.779660] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:33:40 (1756676020) [ 214.816423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.835474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.875244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 217.573903] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:33:42 (1756676022) [ 217.609271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.632547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.672890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 217.865615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 220.520121] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:33:45 (1756676025) [ 220.556472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.575882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.622810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 220.641526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 223.206813] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:33:48 (1756676028) [ 223.243980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.268660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.306851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 223.325739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 225.842771] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:33:51 (1756676031) [ 225.884392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.905581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.947911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 225.966585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 228.320362] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:33:53 (1756676033) [ 228.352848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 228.364061] LNet: 25126:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 228.367256] LNet: 25126:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 228.378664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 228.415828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 228.433650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 230.858634] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:33:56 (1756676036) [ 230.893440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.911558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.944181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 230.974787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 230.986136] LNetError: 25729:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 231.006124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 231.023386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 233.351882] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:33:58 (1756676038) [ 233.386593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.407879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.455362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8aMyLWw0S8 [ 233.491547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8aMyLWw0S8 [ 233.517276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8aMyLWw0S8 [ 235.878844] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:34:01 (1756676041) [ 235.909729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.932923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.967920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 235.986619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 236.009495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 238.337651] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:34:03 (1756676043) [ 238.370350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.388656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.425379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 239.441174] LNet: Message forwarding will be enabled in 2s [ 242.527161] LNet: Message forwarding enabled [ 242.593941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 242.603859] LNet: Message forwarding will stop in 2s to 6s [ 246.809213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 247.498243] LNet: Message forwarding disabled [ 248.069146] LNet: Message forwarding will be enabled in 2s [ 250.600346] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:34:15 (1756676055) [ 250.634569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.239592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 251.272084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 253.704989] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:34:18 (1756676058) [ 253.744977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.766034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.800828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 253.817660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 253.833780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 253.849371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 253.868225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 253.886383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 253.901959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 253.919985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 253.939211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 253.955203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 253.974096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 253.990104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 254.007244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 254.023785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 254.042128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 254.057198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 254.073155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 254.089815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 254.109293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 254.120134] LNetError: 28795:0:(router.c:628:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 256.552955] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:34:21 (1756676061) [ 256.599270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.621143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.667910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 259.135442] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:34:24 (1756676064) [ 259.836029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg443-server [ 260.449753] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:34:25 (1756676065) [ 260.480374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.498216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.523235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 260.535877] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 261.131598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[44-48]@tcp [ 261.138375] LNet: 30507:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 261.146272] LNetError: 30300:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 261.152190] LNetError: 30300:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.45@tcp has gone from up to down [ 262.177804] LNetError: 30300:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 262.181397] LNetError: 30300:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 263.202175] LNetError: 30300:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 263.208032] LNetError: 30300:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 263.496742] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:34:28 (1756676068) [ 264.223359] LNetError: 30508:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.47@tcp has gone from up to down [ 264.230042] LNetError: 30508:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 264.236050] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg443-server [ 264.895034] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:34:30 (1756676070) [ 264.934101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.946646] LNetError: 31434:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 264.957961] LNet: Removed LNI 192.168.204.43@tcp [ 264.969500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.993315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 265.003407] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 265.275363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[44-49/2]@tcp [ 265.286933] LNet: 31554:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 265.291226] LNet: 31554:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 265.298648] LNetError: 31443:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 265.303729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[44-49/2]@tcp [ 265.304087] LNetError: 31443:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 265.304107] LNetError: 31443:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.44@tcp has gone from up to down [ 265.313847] LNetError: 31443:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 267.818770] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:34:33 (1756676073) [ 267.853181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.863032] LNetError: 32141:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 267.871041] LNet: Removed LNI 192.168.204.43@tcp [ 267.881760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.926278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 267.955296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.973873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.004916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.044800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.065424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.091160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 268.125205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.140807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.172497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.205280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.223959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.247822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 268.283372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.300337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.330304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.363480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.380188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.401696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 268.428621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.443610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.472193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.502597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.516422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.537598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 268.572186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.591777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.627131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.666116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.685331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.716930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 268.748652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.767873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.801226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 271.368563] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:34:36 (1756676076) [ 272.081845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg443-server [ 272.768716] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:34:38 (1756676078) [ 273.566534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg443-server [ 274.327608] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:34:39 (1756676079) [ 274.700846] Key type .llcrypt unregistered [ 274.702694] Key type ._llcrypt unregistered [ 277.054039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 277.098026] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 277.099664] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 277.102334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 279.340950] Key type ._llcrypt registered [ 279.342560] Key type .llcrypt registered [ 279.433741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 279.445732] alg: No test for adler32 (adler32-zlib) [ 280.269880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.298714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 280.314048] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 280.614739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 280.626620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 281.492340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 281.501113] LNet: Removed LNI 10.1.2.3@tcp [ 281.548107] LNet: 34255:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 281.552104] LNetError: 34255:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 281.565609] LNet: Removed LNI 192.168.204.43@tcp [ 281.857554] Key type .llcrypt unregistered [ 281.859250] Key type ._llcrypt unregistered [ 285.982991] Key type ._llcrypt registered [ 285.984866] Key type .llcrypt registered [ 286.519417] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:34:51 (1756676091) [ 286.703367] Key type .llcrypt unregistered [ 286.705169] Key type ._llcrypt unregistered [ 288.880554] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 288.930481] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 288.932848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 288.935222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 291.175937] Key type ._llcrypt registered [ 291.177599] Key type .llcrypt registered [ 291.246432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 291.253247] alg: No test for adler32 (adler32-zlib) [ 292.060043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.083668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 292.095394] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 292.388869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 292.402955] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 293.353269] LNet: 35733:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.356602] LNetError: 35733:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.360246] LNet: Removed LNI 192.168.204.43@tcp [ 293.371343] LNet: Removed LNI 10.1.2.3@tcp [ 293.616471] Key type .llcrypt unregistered [ 293.617845] Key type ._llcrypt unregistered [ 297.511676] Key type ._llcrypt registered [ 297.513213] Key type .llcrypt registered [ 298.011145] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:35:03 (1756676103) [ 298.211387] Key type .llcrypt unregistered [ 298.213107] Key type ._llcrypt unregistered [ 300.340616] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 300.406820] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 300.410283] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 300.413301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 302.633123] Key type ._llcrypt registered [ 302.634107] Key type .llcrypt registered [ 302.703717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 302.710376] alg: No test for adler32 (adler32-zlib) [ 303.516545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.541316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 303.555335] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 304.293320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 304.303955] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 305.094164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 305.102617] LNet: Removed LNI 10.1.2.3@tcp [ 305.110260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 305.131670] LNet: Removed LNI 10.1.2.31@tcp [ 305.139412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 305.151115] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 305.417729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 305.427185] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 305.785887] LNet: 37631:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.789921] LNetError: 37631:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 305.793511] LNet: Removed LNI 10.1.2.3@tcp [ 305.801392] LNet: Removed LNI 10.1.2.31@tcp [ 306.051481] Key type .llcrypt unregistered [ 306.053147] Key type ._llcrypt unregistered [ 309.966955] Key type ._llcrypt registered [ 309.968611] Key type .llcrypt registered [ 310.468949] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:35:15 (1756676115) [ 310.608948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 310.616050] alg: No test for adler32 (adler32-zlib) [ 311.433142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.462239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 311.475042] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 311.769528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.781240] LNet: 38575:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.785286] LNetError: 38575:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.794718] LNet: Removed LNI 192.168.204.43@tcp [ 311.804924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.827520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.43@tcp [ 311.839029] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 312.111383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.43@tcp [ 312.126368] LNet: Removed LNI 192.168.204.43@tcp [ 312.397619] Key type .llcrypt unregistered [ 312.399289] Key type ._llcrypt unregistered [ 316.103726] Key type ._llcrypt registered [ 316.105286] Key type .llcrypt registered [ 316.618989] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:35:21 (1756676121) [ 317.294347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg443-server [ 317.904077] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:35:23 (1756676123) [ 318.079481] Key type .llcrypt unregistered [ 318.081144] Key type ._llcrypt unregistered [ 319.971710] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 320.042123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 320.044529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 320.047094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 322.246408] Key type ._llcrypt registered [ 322.247812] Key type .llcrypt registered [ 322.319700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 322.330680] alg: No test for adler32 (adler32-zlib) [ 323.148875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.176939] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 323.430833] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 324.203176] LNet: 40527:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.205706] LNetError: 40527:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.208359] LNet: Removed LNI 192.168.204.43@tcp1 [ 324.221304] LNet: Removed LNI 10.1.2.3@tcp2 [ 324.471416] Key type .llcrypt unregistered [ 324.473042] Key type ._llcrypt unregistered [ 328.223212] Key type ._llcrypt registered [ 328.224765] Key type .llcrypt registered [ 328.747106] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:35:34 (1756676134) [ 328.885923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.893628] alg: No test for adler32 (adler32-zlib) [ 329.706089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.730299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 329.741432] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 330.039327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.047052] LNet: 41482:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.049119] LNetError: 41482:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.062319] LNet: Removed LNI 192.168.204.43@tcp [ 330.070283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.115546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 330.125533] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 332.589293] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:35:37 (1756676137) [ 332.619358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.629109] LNet: 42072:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.632484] LNetError: 42072:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.641496] LNet: Removed LNI 192.168.204.43@tcp [ 332.651598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.675388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 332.686073] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 332.953777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.43 [ 332.963689] LNet: Removed LNI 192.168.204.43@tcp [ 335.127070] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:35:40 (1756676140) [ 335.176567] LNet: 42776:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.179873] LNetError: 42776:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 335.189373] LNet: Removed LNI 192.168.204.43@tcp [ 335.454532] Key type .llcrypt unregistered [ 335.456069] Key type ._llcrypt unregistered [ 337.582319] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 337.689547] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 337.691812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 337.694518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 339.974411] Key type ._llcrypt registered [ 339.975431] Key type .llcrypt registered [ 340.041974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 340.049461] alg: No test for adler32 (adler32-zlib) [ 341.019848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.285775] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:35:48 (1756676148) [ 343.318709] LNet: 43673:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.321792] LNetError: 43673:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 343.331851] LNet: Removed LNI 10.1.2.3@tcp [ 343.583733] Key type .llcrypt unregistered [ 343.585383] Key type ._llcrypt unregistered [ 345.576487] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 345.666873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 345.669340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 345.671948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 347.930704] Key type ._llcrypt registered [ 347.931988] Key type .llcrypt registered [ 347.983096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 347.988950] alg: No test for adler32 (adler32-zlib) [ 348.810803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 350.852340] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:35:56 (1756676156) [ 350.900649] LNet: 44564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.905446] LNetError: 44564:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 350.913364] LNet: Removed LNI 10.1.2.3@tcp [ 351.161533] Key type .llcrypt unregistered [ 351.163085] Key type ._llcrypt unregistered [ 353.153703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 353.248959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 353.251424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 353.254022] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 355.514649] Key type ._llcrypt registered [ 355.516334] Key type .llcrypt registered [ 355.570836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 355.575458] alg: No test for adler32 (adler32-zlib) [ 356.388167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 358.597172] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:36:03 (1756676163) [ 358.633384] LNet: 45448:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.636726] LNetError: 45448:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 358.646495] LNet: Removed LNI 10.1.2.3@tcp [ 358.900776] Key type .llcrypt unregistered [ 358.902547] Key type ._llcrypt unregistered [ 361.040145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 361.139080] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 361.141404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 361.143939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 363.405796] Key type ._llcrypt registered [ 363.407375] Key type .llcrypt registered [ 363.480029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 363.486979] alg: No test for adler32 (adler32-zlib) [ 364.288466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.322244] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 367.211676] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:36:12 (1756676172) [ 367.372706] LNet: 46676:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.376645] LNetError: 46676:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 367.385422] LNet: Removed LNI 10.1.2.3@tcp [ 367.617676] Key type .llcrypt unregistered [ 367.619201] Key type ._llcrypt unregistered [ 369.552173] Key type ._llcrypt registered [ 369.553493] Key type .llcrypt registered [ 369.620600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 369.626437] alg: No test for adler32 (adler32-zlib) [ 370.418858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 370.427123] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 373.329897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 373.565950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 373.584176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 373.602633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 373.612815] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 373.728359] LNet: 47100:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 373.759691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 373.774942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 373.790671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 373.981165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 373.995763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 374.011451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 374.149930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 374.164394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 374.178111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 374.361439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 374.380296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 374.397795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 374.407308] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 374.409677] LNet: Skipped 1 previous similar message [ 374.526041] LNet: 47295:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 374.529548] LNet: 47295:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 374.559644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 374.576171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 374.592060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 374.785434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 374.802189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 374.815979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 374.959499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 374.973872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 374.985529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 375.164702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 375.180831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 375.199886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 375.346132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.358586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.371904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.558409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 375.574031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 375.588799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 375.596870] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 375.600062] LNet: Skipped 2 previous similar messages [ 375.710143] LNet: 47586:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 375.712472] LNet: 47586:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 375.741758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 375.756324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 375.771711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 375.864677] LNet: 47622:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.868322] LNetError: 47622:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.878526] LNet: Removed LNI 192.168.204.43@tcp [ 376.024470] Key type .llcrypt unregistered [ 376.025975] Key type ._llcrypt unregistered [ 380.885974] Key type ._llcrypt registered [ 380.887368] Key type .llcrypt registered [ 381.377667] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:36:26 (1756676186) [ 381.695347] Key type .llcrypt unregistered [ 381.697035] Key type ._llcrypt unregistered [ 383.703521] Key type ._llcrypt registered [ 383.704864] Key type .llcrypt registered [ 383.775772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 383.782984] alg: No test for adler32 (adler32-zlib) [ 384.585104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 384.597953] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 387.360415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 390.472972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.485873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 390.494061] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 390.918098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 390.929692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 1 [ 390.943333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 390.960192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 1 [ 390.974678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 390.989734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 1 [ 391.002783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 391.017511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 1 [ 391.034620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 391.043399] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 391.045946] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 391.137056] LNet: 49066:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 391.158813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 391.169465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 391.183314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 391.346983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 391.360340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 1 [ 391.374890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 391.390165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 1 [ 391.406219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 391.420286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 1 [ 391.434760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 391.449368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 1 [ 391.463309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 391.587497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 391.599364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 391.614753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 391.792304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 391.804819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 1 [ 391.820935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 391.834321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 1 [ 391.848667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 391.861794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 1 [ 391.876162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 391.892112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 1 [ 391.906067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 391.913312] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 391.915659] LNet: Skipped 5 previous similar messages [ 392.016807] LNet: 49284:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 392.018917] LNet: 49284:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 392.041327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 392.053730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 392.065381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 392.234977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 392.249348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 1 [ 392.262885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 392.275396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 1 [ 392.291472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 392.309934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 1 [ 392.327356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 392.341984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 1 [ 392.354803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 392.475784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 392.486079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 392.498139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 392.670127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 392.682757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 1 [ 392.696580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 392.710772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 1 [ 392.726155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 392.743962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 1 [ 392.760272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 392.775264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 1 [ 392.789160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 392.917230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 392.930658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 392.944767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 393.113180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 393.127837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_error -r 1 [ 393.140579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 393.152235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_error -r 1 [ 393.165417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 393.178812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_error -r 1 [ 393.193298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 393.207696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_error -r 1 [ 393.221576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 393.229197] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 393.232731] LNet: Skipped 9 previous similar messages [ 393.343988] LNet: 49612:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 393.347900] LNet: 49612:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 393.371097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 393.382049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 393.395392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 395.489970] LNet: 49682:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.492342] LNetError: 49682:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.494993] LNet: Removed LNI 192.168.204.43@tcp [ 396.517314] LNet: Removed LNI 192.168.204.43@tcp1 [ 396.676562] Key type .llcrypt unregistered [ 396.677691] Key type ._llcrypt unregistered [ 401.763676] Key type ._llcrypt registered [ 401.764679] Key type .llcrypt registered [ 402.209368] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:36:47 (1756676207) [ 402.478339] Key type .llcrypt unregistered [ 402.479407] Key type ._llcrypt unregistered [ 404.232189] Key type ._llcrypt registered [ 404.233929] Key type .llcrypt registered [ 404.287768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 404.292151] alg: No test for adler32 (adler32-zlib) [ 405.090347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 405.101161] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 407.915321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 408.126717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 408.139515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 408.154568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 408.161772] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 408.277236] LNet: 50913:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 408.303382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 408.316907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 408.329142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 408.509085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 408.526122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 408.541744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 408.666113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 408.678952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 408.690216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 408.859572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 408.872303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 408.887174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 408.897894] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 408.901381] LNet: Skipped 1 previous similar message [ 409.001622] LNet: 51108:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 409.004708] LNet: 51108:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 409.028940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 409.041429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 409.052088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 409.133325] LNet: 51144:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.137307] LNetError: 51144:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.146239] LNet: Removed LNI 192.168.204.43@tcp [ 409.280406] Key type .llcrypt unregistered [ 409.281860] Key type ._llcrypt unregistered [ 414.353592] Key type ._llcrypt registered [ 414.355207] Key type .llcrypt registered [ 414.829856] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:37:00 (1756676220) [ 415.126541] Key type .llcrypt unregistered [ 415.128041] Key type ._llcrypt unregistered [ 416.881528] Key type ._llcrypt registered [ 416.883189] Key type .llcrypt registered [ 416.947514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 416.953250] alg: No test for adler32 (adler32-zlib) [ 417.745942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 417.753981] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 420.410837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 423.105479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.119875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 423.129248] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 423.622124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 423.638356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 1 [ 423.657575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 423.673214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 1 [ 423.689397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 423.708478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 1 [ 423.726550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 423.743616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 1 [ 423.761300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 423.771176] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 423.773309] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 423.886458] LNet: 52598:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 423.918279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 423.934694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 423.951392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 423.968885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 424.163401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 424.180039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_error -r 1 [ 424.198604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 424.215559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_error -r 1 [ 424.232428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 424.248379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_error -r 1 [ 424.264550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 424.278328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_error -r 1 [ 424.294123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 424.303832] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 424.306778] LNet: Skipped 2 previous similar messages [ 424.423866] LNet: 52709:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 424.454628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 424.469945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 424.486781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 424.502419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 424.717250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 424.736211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_timeout -r 1 [ 424.754529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 424.770988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_timeout -r 1 [ 424.786491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 424.800745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_timeout -r 1 [ 424.816146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 424.833381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_timeout -r 1 [ 424.851065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 425.011583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 425.027961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 425.045641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 425.061145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 427.291499] LNet: 52892:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.295862] LNetError: 52892:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.299687] LNet: Removed LNI 192.168.204.43@tcp [ 427.310380] LNet: Removed LNI 192.168.204.43@tcp1 [ 427.482565] Key type .llcrypt unregistered [ 427.484431] Key type ._llcrypt unregistered [ 432.802580] Key type ._llcrypt registered [ 432.803757] Key type .llcrypt registered [ 433.271996] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:37:18 (1756676238) [ 433.447586] Key type .llcrypt unregistered [ 433.449078] Key type ._llcrypt unregistered [ 435.226281] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 435.263478] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 435.265810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 435.267951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 437.471130] Key type ._llcrypt registered [ 437.472485] Key type .llcrypt registered [ 437.525468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 437.530026] alg: No test for adler32 (adler32-zlib) [ 438.333193] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 438.346861] LNet: 53975:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.349564] LNetError: 53975:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.356406] LNet: Removed LNI 192.168.204.43@tcp [ 438.579431] Key type .llcrypt unregistered [ 438.581059] Key type ._llcrypt unregistered [ 440.437447] Key type ._llcrypt registered [ 440.438510] Key type .llcrypt registered [ 440.490621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 440.495894] alg: No test for adler32 (adler32-zlib) [ 441.296060] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 441.298118] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 441.310330] LNet: 54244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.312767] LNetError: 54244:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 441.314654] LNet: Removed LNI 192.168.204.43@tcp [ 441.321315] LNet: Removed LNI 10.1.2.3@tcp1 [ 441.538411] Key type .llcrypt unregistered [ 441.540100] Key type ._llcrypt unregistered [ 443.425878] Key type ._llcrypt registered [ 443.427315] Key type .llcrypt registered [ 443.486378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 443.492516] alg: No test for adler32 (adler32-zlib) [ 444.297392] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 444.299325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 444.310658] LNet: 54512:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.312728] LNetError: 54512:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 444.315089] LNet: Removed LNI 192.168.204.43@tcp [ 444.321497] LNet: Removed LNI 10.1.2.3@tcp [ 444.569366] Key type .llcrypt unregistered [ 444.570471] Key type ._llcrypt unregistered [ 446.373687] Key type ._llcrypt registered [ 446.374903] Key type .llcrypt registered [ 446.435098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 446.442347] alg: No test for adler32 (adler32-zlib) [ 447.243318] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 447.244768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 447.257163] LNet: 54788:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.260344] LNetError: 54788:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.263729] LNet: Removed LNI 192.168.204.43@tcp [ 447.275304] LNet: Removed LNI 10.1.2.3@tcp [ 447.509631] Key type .llcrypt unregistered [ 447.511456] Key type ._llcrypt unregistered [ 449.358903] Key type ._llcrypt registered [ 449.359915] Key type .llcrypt registered [ 449.410748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 449.416530] alg: No test for adler32 (adler32-zlib) [ 450.221958] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 450.223620] LNetError: 55047:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 450.226748] LNetError: Error -100 starting up LNI tcp [ 450.234335] LNet: Removed LNI 192.168.204.43@tcp [ 450.463467] Key type .llcrypt unregistered [ 450.465271] Key type ._llcrypt unregistered [ 453.789596] Key type ._llcrypt registered [ 453.790506] Key type .llcrypt registered [ 454.225097] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:37:39 (1756676259) [ 454.483201] Key type .llcrypt unregistered [ 454.484164] Key type ._llcrypt unregistered [ 456.133144] Key type ._llcrypt registered [ 456.134232] Key type .llcrypt registered [ 456.203405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 456.210820] alg: No test for adler32 (adler32-zlib) [ 457.011563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 457.020975] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 459.709602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 459.918384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 459.930866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 459.943731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 459.950721] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 460.047767] LNet: 56284:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 460.069619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 460.079826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 460.157863] LNet: 56318:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.160424] LNetError: 56318:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 460.168245] LNet: Removed LNI 192.168.204.43@tcp [ 460.293444] Key type .llcrypt unregistered [ 460.294515] Key type ._llcrypt unregistered [ 465.462097] Key type ._llcrypt registered [ 465.463132] Key type .llcrypt registered [ 465.539080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 465.546065] alg: No test for adler32 (adler32-zlib) [ 466.357897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 466.369402] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 469.070829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 471.475087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.486986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 471.495732] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 471.993450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 472.008901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e network_timeout -r 1 [ 472.026488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 472.042162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e network_timeout -r 1 [ 472.057785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 472.074775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e network_timeout -r 1 [ 472.091102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 472.106973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e network_timeout -r 1 [ 472.120653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 472.128776] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 472.241668] LNet: 57176:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 472.267415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 472.281300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 474.241143] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 474.343696] LNet: 57244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.346366] LNetError: 57244:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.348884] LNet: Removed LNI 192.168.204.43@tcp [ 474.357053] LNet: Removed LNI 192.168.204.43@tcp1 [ 474.505375] Key type .llcrypt unregistered [ 474.506817] Key type ._llcrypt unregistered [ 479.685524] Key type ._llcrypt registered [ 479.686762] Key type .llcrypt registered [ 480.146625] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:38:05 (1756676285) [ 480.269651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 480.274743] alg: No test for adler32 (adler32-zlib) [ 481.076886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.098250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 481.107692] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 481.346049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 481.355416] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 481.649972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 481.675061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 481.693371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 481.707376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 481.721214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 481.733775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 481.748892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 481.758974] LNetError: 58332:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 482.143247] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 482.143268] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 484.191284] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 484.910317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 488.287342] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 489.100899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 489.112240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 489.137246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.144586] LNet: 58450:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.147774] LNetError: 58450:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.150299] LNet: Removed LNI 192.168.204.43@tcp [ 489.155321] LNet: Removed LNI 192.168.204.43@tcp1 [ 489.163133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.184253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.194428] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 489.396625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 489.403116] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 489.643293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 489.654305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 489.675027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 489.689580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 489.702115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 489.713070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 489.725487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 489.738410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 489.746473] LNetError: 58716:0:(api-ni.c:4315:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 492.255341] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 492.258368] LNet: Skipped 1 previous similar message [ 492.894588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 496.351365] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 501.215171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 501.228357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 501.241280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 503.387828] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:38:28 (1756676308) [ 503.425219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.434320] LNet: 59440:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.436556] LNetError: 59440:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.438836] LNetError: 59440:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 503.440992] LNet: Removed LNI 192.168.204.43@tcp [ 503.447757] LNet: Removed LNI 192.168.204.43@tcp1 [ 503.457325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.479668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 503.490532] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 503.742616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 503.750422] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 504.023859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 504.046911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 504.057251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 504.070538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 504.083701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 504.097195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 504.109369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 0 [ 504.144142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 504.479169] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.43@tcp [ 505.503285] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 505.505257] LNet: Skipped 1 previous similar message [ 520.550697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 520.593158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 520.602295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 520.612892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 500 [ 520.643662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 527.859362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 527.880878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.888292] LNet: 59951:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.890348] LNetError: 59951:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.892320] LNetError: 59951:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 527.894889] LNet: Removed LNI 192.168.204.43@tcp [ 527.908345] LNet: Removed LNI 192.168.204.43@tcp1 [ 527.916136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.935249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 527.946532] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 528.184412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 528.191861] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 528.479330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 528.494879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 528.519594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 528.531674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 528.546215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 528.564358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 528.577959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 528.589323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 0 [ 529.951391] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 529.953836] LNet: Skipped 3 previous similar messages [ 530.735282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 540.126232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 540.138139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 540.151845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.534067] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:39:07 (1756676347) [ 542.677552] LNet: 61000:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.680099] LNetError: 61000:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.684051] LNetError: 61000:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 542.686284] LNet: Removed LNI 192.168.204.43@tcp [ 542.692579] LNet: Removed LNI 192.168.204.43@tcp1 [ 542.936728] Key type .llcrypt unregistered [ 542.937547] Key type ._llcrypt unregistered [ 544.802381] Key type ._llcrypt registered [ 544.804305] Key type .llcrypt registered [ 544.871453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.877463] alg: No test for adler32 (adler32-zlib) [ 545.680642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 545.692218] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 548.952135] Lustre: DEBUG MARKER: Initial discovery [ 548.966151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 549.778190] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 549.795743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 549.811234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 549.819889] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 549.828781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 549.845517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.143@tcp -r 1 -m GET -l 3 [ 549.861469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.143@tcp -r 1 -m GET -e local_error [ 549.877505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.143@tcp -r 1 -m PUT -l 6 [ 550.351130] Lustre: DEBUG MARKER: Force oleg443-client.virtnet to discover 192.168.204.143@tcp (in background) [ 550.371332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 550.881152] Lustre: DEBUG MARKER: Force oleg443-server to discover 192.168.204.43@tcp [ 551.680935] Lustre: DEBUG MARKER: Wait for 61749 [ 553.439269] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 553.887747] Lustre: DEBUG MARKER: Finished wait on 61749 [ 553.918342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 553.936234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 554.035236] LNet: 62204:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.038618] LNetError: 62204:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.050392] LNet: Removed LNI 192.168.204.43@tcp [ 554.212396] Key type .llcrypt unregistered [ 554.214230] Key type ._llcrypt unregistered [ 559.655966] Key type ._llcrypt registered [ 559.657059] Key type .llcrypt registered [ 560.121690] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:39:25 (1756676365) [ 560.305376] Key type .llcrypt unregistered [ 560.306622] Key type ._llcrypt unregistered [ 562.471544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 562.539550] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 562.542076] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 562.544279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 564.757903] Key type ._llcrypt registered [ 564.759453] Key type .llcrypt registered [ 564.828314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 564.834455] alg: No test for adler32 (adler32-zlib) [ 565.642269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.666367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 565.674757] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 565.960097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 565.969294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 568.942468] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:39:34 (1756676374) [ 568.979915] LNet: 64264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.983579] LNetError: 64264:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 568.987155] LNet: Removed LNI 192.168.204.43@tcp [ 568.994085] LNet: Removed LNI 10.1.2.3@tcp [ 569.250404] Key type .llcrypt unregistered [ 569.251588] Key type ._llcrypt unregistered [ 571.305166] Key type ._llcrypt registered [ 571.307092] Key type .llcrypt registered [ 571.380045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 571.386239] alg: No test for adler32 (adler32-zlib) [ 572.190309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.213351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 572.224849] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 572.479786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 572.487532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 572.841421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.43@tcp [ 573.855200] LNetError: Unexpected error -22 connecting to 192.168.204.43@tcp at host 192.168.204.43:988 [ 575.089090] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:39:40 (1756676380) [ 575.118561] LNet: 65384:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.120762] LNetError: 65384:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.124153] LNet: Removed LNI 192.168.204.43@tcp [ 575.132601] LNet: Removed LNI 10.1.2.3@tcp [ 575.349271] Key type .llcrypt unregistered [ 575.350231] Key type ._llcrypt unregistered [ 577.156561] Key type ._llcrypt registered [ 577.157802] Key type .llcrypt registered [ 577.219047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.224016] alg: No test for adler32 (adler32-zlib) [ 578.025720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.044705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 578.055059] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 578.271548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 578.278102] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 578.547856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.43@tcp1 --nid 192.168.204.43@tcp2 [ 579.768850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp1 192.168.204.43@tcp2 [ 579.775790] LNetError: 66215:0:(lib-move.c:2262:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.43@tcp1 to 192.168.204.43@tcp2 - Route table may be misconfigured [ 579.784418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp2 192.168.204.43@tcp1 [ 581.628142] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:39:47 (1756676387) [ 581.657432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.665899] LNet: 66783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.668481] LNetError: 66783:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.671284] LNet: Removed LNI 192.168.204.43@tcp1 [ 582.693491] LNet: Removed LNI 192.168.204.43@tcp2 [ 582.701491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.720651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 582.730241] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 582.957229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 582.964605] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 583.228498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 583.241058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.43@tcp [ 583.253502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp1 192.168.204.43@tcp1 [ 583.269382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.43@tcp -d 192.168.204.43@tcp -e network_timeout [ 583.284144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -e network_timeout [ 583.297960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -e network_timeout [ 583.312719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -e network_timeout [ 583.326548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 583.334064] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 583.340904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 583.370019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 583.401551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 583.711149] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 585.361705] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:39:50 (1756676390) [ 585.392222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.399918] LNet: 67638:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.402562] LNetError: 67638:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.405139] LNetError: 67638:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 585.408030] LNet: Removed LNI 192.168.204.43@tcp [ 586.469473] LNet: Removed LNI 192.168.204.43@tcp1 [ 586.476341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.504128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 586.586172] LNet: 67678:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.733515] Key type .llcrypt unregistered [ 586.734661] Key type ._llcrypt unregistered [ 590.196542] Key type ._llcrypt registered [ 590.197613] Key type .llcrypt registered [ 590.609665] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:39:56 (1756676396) [ 590.712569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 590.717052] alg: No test for adler32 (adler32-zlib) [ 591.515139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.958839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 592.501058] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:39:57 (1756676397) [ 592.538608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.546582] LNet: 68770:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.555753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.573522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.580804] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 592.799112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 592.806822] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 593.067194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 593.079437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp1 [ 593.090221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp1 [ 595.013601] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:40:00 (1756676400) [ 595.582218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg443-server [ 596.098682] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:40:01 (1756676401) [ 596.702032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg443-server [ 597.274348] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:40:02 (1756676402) [ 597.840366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg443-server [ 598.396534] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:40:03 (1756676403) [ 598.982225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg443-server [ 599.526181] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:40:04 (1756676404) [ 600.084351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg443-server [ 600.598465] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:40:06 (1756676406) [ 601.175434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg443-server [ 601.709267] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:40:07 (1756676407) [ 602.278203] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg443-server [ 602.799887] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:40:08 (1756676408) [ 603.359515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg443-server [ 603.913597] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:40:09 (1756676409) [ 604.487690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg443-server [ 605.040800] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:40:10 (1756676410) [ 605.068291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.077380] LNet: 72631:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.080533] LNetError: 72631:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.082643] LNet: Removed LNI 192.168.204.43@tcp [ 606.117520] LNet: Removed LNI 192.168.204.43@tcp1 [ 606.126268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.149114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 606.157429] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 606.388284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 606.418064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 606.477689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.485590] LNet: 72780:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.487943] LNetError: 72780:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.490263] LNetError: 72780:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 606.502286] LNet: Removed LNI 192.168.204.43@tcp [ 606.509165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.528845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 606.539204] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 606.784808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 606.847877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.909284] LNet: Removed LNI 192.168.204.43@tcp [ 607.916922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.937802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.946866] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 608.167726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 608.194337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 608.257799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.265485] LNetError: 73067:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.268263] LNetError: 73067:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 608.274497] LNet: Removed LNI 192.168.204.43@tcp [ 608.281246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.304208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 608.312712] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 608.533353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 608.588701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.595633] LNet: 73209:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.597866] LNet: 73209:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 609.637331] LNet: Removed LNI 192.168.204.43@tcp [ 609.645395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.664154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.672639] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 609.924661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 609.952775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 611.056148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.065825] LNetError: 73362:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.069553] LNetError: 73362:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 611.082400] LNet: Removed LNI 192.168.204.43@tcp [ 611.089756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.108748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 611.121434] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 611.374337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 612.487616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.541375] LNet: Removed LNI 192.168.204.43@tcp [ 613.549924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.571563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 613.582601] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 613.831946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 613.860700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 613.935487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.943088] LNet: 73657:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.945432] LNet: 73657:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 615.013305] LNet: Removed LNI 192.168.204.43@tcp [ 615.021368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.043419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 615.053699] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 615.307523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 615.384923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.393485] LNetError: 73799:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.395908] LNetError: 73799:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 615.404434] LNet: Removed LNI 192.168.204.43@tcp [ 615.412600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.431947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.443300] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 615.688324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 615.720947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 615.800694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.869348] LNet: Removed LNI 192.168.204.43@tcp [ 616.878828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.901539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 616.912347] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 617.176788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 617.268778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.341348] LNet: Removed LNI 192.168.204.43@tcp [ 618.349805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.374673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.386282] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 618.655405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 618.685907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 618.772379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.813366] LNet: Removed LNI 192.168.204.43@tcp [ 619.821495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.843254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 619.854853] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 620.120501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 622.296374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.304551] LNet: 74391:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.308318] LNet: 74391:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 623.337333] LNet: Removed LNI 192.168.204.43@tcp [ 623.345134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.365403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.375174] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 623.621482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 623.647626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.656969] LNetError: 74521:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.659194] LNetError: 74521:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 623.670331] LNet: Removed LNI 192.168.204.43@tcp [ 623.677947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.699859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.710448] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 624.000849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 626.436456] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:40:31 (1756676431) [ 626.466320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.482126] LNet: Removed LNI 192.168.204.43@tcp [ 626.492291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.522287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 626.533290] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 626.950442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.963296] LNet: Removed LNI 192.168.204.43@tcp231 [ 626.971243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.996131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 627.005981] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 627.007822] LNet: 75364:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 627.149778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 627.168655] LNet: Removed LNI 192.168.204.43@tcp231 [ 627.175356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 627.185149] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 627.544525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.561287] LNet: Removed LNI 192.168.204.43@tcp231 [ 627.568977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.593352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 627.601717] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/0] [ 627.603566] LNet: 75513:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 627.732137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 627.749481] LNet: Removed LNI 192.168.204.43@tcp231 [ 627.755538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 627.764202] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/0] [ 628.172368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.184558] LNet: Removed LNI 192.168.204.43@tcp231 [ 628.192363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.217152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 628.227766] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/60] [ 628.369819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 628.386523] LNet: Removed LNI 192.168.204.43@tcp231 [ 628.395508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 628.406241] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/60] [ 628.810378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.826311] LNet: Removed LNI 192.168.204.43@tcp231 [ 628.835401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.862911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 628.874312] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 628.876891] LNet: 75809:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 628.880384] LNet: 75809:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 629.028786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 629.045619] LNet: Removed LNI 192.168.204.43@tcp231 [ 629.052560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 629.062532] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 631.583588] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:40:36 (1756676436) [ 631.618418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.635800] LNet: Removed LNI 192.168.204.43@tcp231 [ 631.646565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.676942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 631.689535] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 631.951170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 631.967172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 631.982740] LNet: Removed LNI 192.168.204.43@tcp232 [ 631.990802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 632.002115] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 632.275831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 632.293479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 632.315388] LNet: Removed LNI 192.168.204.43@tcp232 [ 632.322383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 632.332269] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 632.589984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 632.608973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 632.629583] LNet: Removed LNI 192.168.204.43@tcp232 [ 632.637512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 632.649210] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 632.911604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 632.928852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 632.946626] LNet: Removed LNI 192.168.204.43@tcp232 [ 632.953681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 635.177973] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:40:40 (1756676440) [ 635.583125] Key type .llcrypt unregistered [ 635.584344] Key type ._llcrypt unregistered [ 637.851619] Key type ._llcrypt registered [ 637.852888] Key type .llcrypt registered [ 637.917370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.923423] alg: No test for adler32 (adler32-zlib) [ 638.731534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 638.743337] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 641.472819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 643.897689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.912903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 643.922040] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 644.449932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 2 [ 644.465261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 2 [ 644.484129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 2 [ 644.498742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 2 [ 644.511304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_interrupt -r 2 [ 644.524067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 2 [ 644.536717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_interrupt -r 2 [ 644.550962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 2 [ 644.566198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 644.574549] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 644.576992] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 644.687048] LNet: 78276:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 644.713154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 644.726795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 644.916576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 2 [ 644.930814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 2 [ 644.942335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 2 [ 644.953978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 2 [ 644.970155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_dropped -r 2 [ 644.983922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 2 [ 644.997498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_dropped -r 2 [ 645.011584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 2 [ 645.026128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 645.165162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 645.179830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 645.400791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 2 [ 645.414747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 2 [ 645.427688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 2 [ 645.443247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 2 [ 645.463757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_aborted -r 2 [ 645.480902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 2 [ 645.497455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_aborted -r 2 [ 645.514086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 2 [ 645.526396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 645.533829] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 645.535887] LNet: Skipped 2 previous similar messages [ 645.657915] LNet: 78492:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 645.661347] LNet: 78492:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 645.690069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 645.706450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 645.895883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 2 [ 645.913951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 2 [ 645.927263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 2 [ 645.942269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 2 [ 645.957844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_no_route -r 2 [ 645.971802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 2 [ 645.986857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_no_route -r 2 [ 646.002929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 2 [ 646.019860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 646.173201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.188703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 646.392971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 2 [ 646.407516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 2 [ 646.424299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 2 [ 646.439468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 2 [ 646.455824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_timeout -r 2 [ 646.475340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 2 [ 646.491462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_timeout -r 2 [ 646.505313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 2 [ 646.521608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 646.676594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.691759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 646.887403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 2 [ 646.902256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 2 [ 646.915712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 2 [ 646.930714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 2 [ 646.946857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_dropped -r 2 [ 646.961802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 2 [ 646.978116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_dropped -r 2 [ 646.996191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 2 [ 647.010945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 647.018740] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 647.020861] LNet: Skipped 3 previous similar messages [ 647.022657] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 647.136763] LNet: 78815:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 647.139611] LNet: 78815:0:(api-ni.c:338:recovery_interval_set()) Skipped 2 previous similar messages [ 647.166592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.180850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 649.378943] LNet: 78885:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.381653] LNetError: 78885:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.384584] LNet: Removed LNI 192.168.204.43@tcp [ 649.391376] LNet: Removed LNI 192.168.204.43@tcp1 [ 649.550706] Key type .llcrypt unregistered [ 649.552451] Key type ._llcrypt unregistered [ 654.714066] Key type ._llcrypt registered [ 654.715363] Key type .llcrypt registered [ 655.202594] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:41:00 (1756676460) [ 655.329036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.334964] alg: No test for adler32 (adler32-zlib) [ 656.140070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.160498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.171170] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 656.425744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 656.450549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 656.467064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 656.494415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 656.513846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 656.531777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 658.638637] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:41:04 (1756676464) [ 658.668637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.677449] LNet: 80432:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.679629] LNetError: 80432:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.689686] LNet: Removed LNI 192.168.204.43@tcp [ 658.701137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.721980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.730789] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 661.066757] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:41:06 (1756676466) [ 661.576978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 662.101302] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:41:07 (1756676467) [ 662.255527] LNet: 81471:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.259446] LNetError: 81471:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.270900] LNet: Removed LNI 192.168.204.43@tcp [ 662.508329] Key type .llcrypt unregistered [ 662.509272] Key type ._llcrypt unregistered [ 664.477286] Key type ._llcrypt registered [ 664.478405] Key type .llcrypt registered [ 664.556329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.562662] alg: No test for adler32 (adler32-zlib) [ 665.363444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 665.372441] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 668.214853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 669.665607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.143@tcp [ 669.776473] LNet: 81881:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.779677] LNetError: 81881:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.792404] LNet: Removed LNI 192.168.204.43@tcp [ 669.937276] Key type .llcrypt unregistered [ 669.938415] Key type ._llcrypt unregistered [ 673.573065] Key type ._llcrypt registered [ 673.574386] Key type .llcrypt registered [ 674.041343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 674.585214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 675.123187] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:41:20 (1756676480) [ 675.245120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.250827] alg: No test for adler32 (adler32-zlib) [ 676.051986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.086366] LNet: 82964:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.328249] Key type .llcrypt unregistered [ 676.329433] Key type ._llcrypt unregistered [ 678.304930] Key type ._llcrypt registered [ 678.306376] Key type .llcrypt registered [ 678.365299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 678.370920] alg: No test for adler32 (adler32-zlib) [ 679.177194] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 679.178561] LNet: 83215:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 679.187271] LNetError: 83227:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 679.193950] LNetError: 83227:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.204.44@tcp has gone from up to down [ 679.352970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.463133] LNetError: 83227:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.204.47@tcp has gone from up to down [ 682.470695] LNetError: 83227:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 682.476081] LNet: 83273:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.478426] LNetError: 83273:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.488535] LNet: Removed LNI 192.168.204.43@tcp [ 682.500791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.524611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.535756] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 682.538301] LNet: 83281:0:(router.c:732:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 682.542198] LNet: 83281:0:(router.c:732:lnet_add_route()) Skipped 4 previous similar messages [ 682.546612] LNetError: 83281:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.44@tcp is a gateway. Can not delete it [ 682.549240] LNetError: 83281:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.204.44@tcp is a router. Can not be deleted [ 682.553876] LNetError: 83278:0:(router.c:397:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 682.559497] LNetError: 83278:0:(router.c:397:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 682.562336] LNet: 83281:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 684.754935] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:41:30 (1756676490) [ 685.343415] LNetError: 83284:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) route to o2ib through 192.168.204.48@tcp has gone from up to down [ 685.349027] LNetError: 83284:0:(lib-lnet.h:1314:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 685.402175] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg443-server [ 685.963376] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:41:31 (1756676491) [ 686.619807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg443-server [ 687.197082] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:41:32 (1756676492) [ 687.790475] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 689.447158] Key type lgssc registered [ 689.892358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 697.726102] Key type lgssc unregistered [ 697.894497] LNet: 85695:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.897879] LNetError: 85695:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.906612] LNet: Removed LNI 192.168.204.43@tcp [ 698.134584] Key type .llcrypt unregistered [ 698.135602] Key type ._llcrypt unregistered [ 703.299395] Key type ._llcrypt registered [ 703.300490] Key type .llcrypt registered [ 703.785555] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:41:49 (1756676509) [ 703.964384] Key type .llcrypt unregistered [ 703.966046] Key type ._llcrypt unregistered [ 705.898629] Key type ._llcrypt registered [ 705.899679] Key type .llcrypt registered [ 705.968288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.975869] alg: No test for adler32 (adler32-zlib) [ 706.784049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.816506] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 706.817839] LNet: *** cfs_fail_loc=e003, val=100*** [ 707.077011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 707.085994] Ping reply received with mapping: [ 707.087500] found metadata reply NI[192.168.204.43@tcp] [ 707.088905] 2b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 707.090405] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 707.091839] 5a 5a 5a 5a 5a 5a 5a 5a [ 707.129845] LNet: 86837:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.133556] LNetError: 86837:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.151321] LNet: Removed LNI 192.168.204.43@tcp [ 707.393294] Key type .llcrypt unregistered [ 707.394595] Key type ._llcrypt unregistered [ 710.864609] Key type ._llcrypt registered [ 710.866405] Key type .llcrypt registered [ 711.359775] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:41:56 (1756676516) [ 711.476720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 711.481839] alg: No test for adler32 (adler32-zlib) [ 712.354405] LNetError: 87676:0:(api-ni.c:2709:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 712.504353] Key type .llcrypt unregistered [ 712.505900] Key type ._llcrypt unregistered [ 714.221708] Key type ._llcrypt registered [ 714.222750] Key type .llcrypt registered [ 714.688816] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:42:00 (1756676520) [ 714.873248] Key type .llcrypt unregistered [ 714.874324] Key type ._llcrypt unregistered [ 716.626223] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 716.628816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 718.791628] Key type ._llcrypt registered [ 718.793431] Key type .llcrypt registered [ 718.863315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 718.870712] alg: No test for adler32 (adler32-zlib) [ 719.675419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.698924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 723.417800] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:42:08 (1756676528) [ 723.474189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.483650] LNet: 89444:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.486254] LNetError: 89444:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 723.494886] LNet: Removed LNI 10.1.2.3@tcp [ 723.502901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.778533] Key type .llcrypt unregistered [ 723.779592] Key type ._llcrypt unregistered [ 725.804579] Key type ._llcrypt registered [ 725.805707] Key type .llcrypt registered [ 725.868558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.874781] alg: No test for adler32 (adler32-zlib) [ 726.685420] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 726.712331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 726.778174] LNet: 89727:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.780468] LNetError: 89727:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.791468] LNet: Removed LNI 192.168.204.43@tcp [ 727.045407] Key type .llcrypt unregistered [ 727.046622] Key type ._llcrypt unregistered [ 730.697601] Key type ._llcrypt registered [ 730.698575] Key type .llcrypt registered [ 731.144146] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:42:16 (1756676536) [ 731.336282] Key type .llcrypt unregistered [ 731.337496] Key type ._llcrypt unregistered [ 733.262461] Key type ._llcrypt registered [ 733.264088] Key type .llcrypt registered [ 733.334801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.339906] alg: No test for adler32 (adler32-zlib) [ 736.282873] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:42:21 (1756676541) [ 736.318462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.333835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.364216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.377306] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 736.633744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.648988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 736.663583] LNet: Removed LNI 192.168.204.43@tcp [ 736.746447] LNet: 91457:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.907406] Key type .llcrypt unregistered [ 736.909325] Key type ._llcrypt unregistered [ 740.905331] Key type ._llcrypt registered [ 740.907015] Key type .llcrypt registered [ 741.410959] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:42:26 (1756676546) [ 741.541468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.547302] alg: No test for adler32 (adler32-zlib) [ 742.353442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.378051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.386772] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 742.663632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 742.682938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.43@tcp [ 742.691459] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 742.693414] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 742.695207] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 742, 0, 0 [ 742.698750] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 742.700833] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 742.702863] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 742.704376] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 742, 0, 0 [ 742.707026] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 742.708174] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 742.710018] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 742.712092] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 742, 0, 792 [ 742.714193] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 742.715822] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 742.717709] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 742.719186] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 742, 0, 792 [ 742.721718] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 744.939692] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:42:30 (1756676550) [ 745.089855] LNet: 92988:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.092771] LNetError: 92988:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.101348] LNet: Removed LNI 192.168.204.43@tcp [ 745.346400] Key type .llcrypt unregistered [ 745.348295] Key type ._llcrypt unregistered [ 747.170362] Key type ._llcrypt registered [ 747.171422] Key type .llcrypt registered [ 747.244154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 747.250730] alg: No test for adler32 (adler32-zlib) [ 748.055674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 748.066224] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 750.771950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 753.423930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.439413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 753.449053] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 753.797766] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 753.862159] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 753.863396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.864672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.021638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 756.029646] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 759.501343] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:42:44 (1756676564) [ 759.532951] LNet: 94475:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.535127] LNetError: 94475:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.537689] LNet: Removed LNI 192.168.204.43@tcp [ 760.543147] LNet: Removed LNI 192.168.204.43@tcp1 [ 760.544475] LNetError: 94475:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface test1pg not found [ 760.548388] LNetError: 94475:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 760.558597] LNet: Removed LNI 10.1.2.3@tcp99 [ 760.808660] Key type .llcrypt unregistered [ 760.810567] Key type ._llcrypt unregistered [ 762.712797] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.715122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 764.889449] Key type ._llcrypt registered [ 764.890467] Key type .llcrypt registered [ 764.956528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.962308] alg: No test for adler32 (adler32-zlib) [ 765.771797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.791956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.801204] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 766.043198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 766.050208] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 766.790401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 766.807392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 766.825109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 766.833464] LNetError: 95215:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 766.848346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 766.871111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 767.018915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 767.046416] LNet: 95235:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.049512] LNetError: 95235:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.052716] LNet: Removed LNI 192.168.204.43@tcp [ 767.057816] LNet: Removed LNI 10.1.2.3@tcp [ 767.321596] Key type .llcrypt unregistered [ 767.322908] Key type ._llcrypt unregistered [ 770.806967] Key type ._llcrypt registered [ 770.808017] Key type .llcrypt registered [ 771.290038] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:42:56 (1756676576) [ 771.437286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 771.443576] alg: No test for adler32 (adler32-zlib) [ 772.259487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.280460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.290878] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 772.545744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 772.573186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg443-client@tcp [ 774.682445] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:43:00 (1756676580) [ 774.712366] LNet: 96752:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.715564] LNetError: 96752:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.723655] LNet: Removed LNI 192.168.204.43@tcp [ 774.981386] Key type .llcrypt unregistered [ 774.982753] Key type ._llcrypt unregistered [ 776.922772] Key type ._llcrypt registered [ 776.924633] Key type .llcrypt registered [ 776.986877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.993675] alg: No test for adler32 (adler32-zlib) [ 777.793272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 777.801552] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 777.814869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 779.914327] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:43:05 (1756676585) [ 780.411720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 780.950581] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:43:06 (1756676586) [ 781.471279] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 782.042128] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:43:07 (1756676587) [ 782.076929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.087946] LNet: 98166:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.092400] LNetError: 98166:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.103409] LNet: Removed LNI 192.168.204.43@tcp [ 782.116189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.137862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.147534] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 782.424204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 782.445405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 784.621628] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:43:10 (1756676590) [ 784.658745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.667041] LNet: 98875:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.669736] LNetError: 98875:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.678465] LNet: Removed LNI 192.168.204.43@tcp [ 784.685413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.704343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 784.715236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 784.928334] Key type .llcrypt unregistered [ 784.929380] Key type ._llcrypt unregistered [ 788.194657] Key type ._llcrypt registered [ 788.195735] Key type .llcrypt registered [ 788.632103] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:43:13 (1756676593) [ 788.747121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.752532] alg: No test for adler32 (adler32-zlib) [ 789.555265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.574092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.583267] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 789.814405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 789.834754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 791.933232] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:43:17 (1756676597) [ 791.964130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.974064] LNet: 100417:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.977467] LNetError: 100417:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.985589] LNet: Removed LNI 192.168.204.43@tcp [ 791.995687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.020215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 792.033886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 794.065417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 794.598080] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:43:19 (1756676599) [ 794.635399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.645674] LNet: 101139:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.658901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.760620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 794.770390] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 794.996279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 795.004181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.291307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 795.304216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 795.325220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.43@tcp -m PUT -e local_timeout -r 1 [ 795.346961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 795.367542] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.369552] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.442913] LNetError: 101175:0:(lib-move.c:4945:lnet_parse()) 192.168.204.43@tcp, src 192.168.204.43@tcp: Dropping PUT (error -108 looking up sender) [ 796.447553] LNet: 101449:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.450895] LNetError: 101449:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.453420] LNet: Removed LNI 192.168.204.43@tcp [ 796.458275] LNet: Removed LNI 10.1.2.3@tcp [ 796.602088] Key type .llcrypt unregistered [ 796.603315] Key type ._llcrypt unregistered [ 799.983564] Key type ._llcrypt registered [ 799.984535] Key type .llcrypt registered [ 800.422285] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:43:25 (1756676605) [ 800.546521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 800.551922] alg: No test for adler32 (adler32-zlib) [ 801.353652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.483860] LNet: 102297:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.688456] Key type .llcrypt unregistered [ 801.689517] Key type ._llcrypt unregistered [ 803.432534] Key type ._llcrypt registered [ 803.433864] Key type .llcrypt registered [ 803.494829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 803.500703] alg: No test for adler32 (adler32-zlib) [ 804.305378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 804.317858] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 807.030143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 807.088545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 807.315391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.143@tcp [ 807.391105] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 807.424523] LNet: 102733:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 807.445824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 807.457464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 807.666684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 807.781230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 807.791423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 807.976344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.143@tcp [ 808.071466] LNet: 102943:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 808.073457] LNet: 102943:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 808.091936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 808.104649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 808.294874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 808.428622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 808.442156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 808.549121] LNet: 103089:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.551206] LNetError: 103089:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.560333] LNet: Removed LNI 192.168.204.43@tcp [ 808.709945] Key type .llcrypt unregistered [ 808.711495] Key type ._llcrypt unregistered [ 814.048447] Key type ._llcrypt registered [ 814.049474] Key type .llcrypt registered [ 814.500190] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:43:39 (1756676619) [ 814.615677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.620777] alg: No test for adler32 (adler32-zlib) [ 815.425206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.564431] LNet: 103972:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.814336] Key type .llcrypt unregistered [ 815.815513] Key type ._llcrypt unregistered [ 817.664190] Key type ._llcrypt registered [ 817.665214] Key type .llcrypt registered [ 817.738615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.744151] alg: No test for adler32 (adler32-zlib) [ 818.560884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 818.571232] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 821.434717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 824.208885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.224733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 824.234189] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 824.589708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 824.796199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.143@tcp [ 824.915434] LNet: 104606:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 824.948106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 824.961186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 825.182752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 825.325935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 825.339952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 825.559867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.143@tcp [ 825.675891] LNet: 104816:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 825.678368] LNet: 104816:0:(api-ni.c:338:recovery_interval_set()) Skipped 1 previous similar message [ 825.701858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 825.714111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 825.935562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 826.078605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 826.092255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 828.325256] LNet: 104997:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.327415] LNetError: 104997:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.331514] LNet: Removed LNI 192.168.204.43@tcp [ 828.337341] LNet: Removed LNI 192.168.204.43@tcp1 [ 828.509494] Key type .llcrypt unregistered [ 828.511048] Key type ._llcrypt unregistered [ 833.832604] Key type ._llcrypt registered [ 833.833917] Key type .llcrypt registered [ 834.312357] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 711 sec =========== 17:43:59 (1756676639) [ 834.543531] Key type .llcrypt unregistered [ 834.544711] Key type ._llcrypt unregistered [ 838.520504] Key type ._llcrypt registered [ 838.521712] Key type .llcrypt registered [ 838.692325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.702892] alg: No test for adler32 (adler32-zlib) [ 839.547723] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 839.622475] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 841.207159] Key type lgssc registered [ 841.568265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.210405] Lustre: Mounted lustre-client [ 864.543338] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 871.760949] LustreError: 108208:0:(lov_obd.c:784:lov_cleanup()) lustre-clilov-ffff991b43b3c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 871.767430] LustreError: 108208:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 871.780241] Lustre: Unmounted lustre-client [ 883.584208] Key type lgssc unregistered [ 883.709391] LNet: 108689:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.711581] LNetError: 108689:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.719643] LNet: Removed LNI 192.168.204.43@tcp [ 883.956140] Key type .llcrypt unregistered [ 883.957662] Key type ._llcrypt unregistered