[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483798923 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002016] Switched APIC routing to physical x2apic. [ 0.003021] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009178] LSM: Security Framework initializing [ 0.010061] Yama: becoming mindful. [ 0.011050] SELinux: Initializing. [ 0.012103] *** VALIDATE selinux *** [ 0.021491] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026441] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029155] *** VALIDATE tmpfs *** [ 0.030476] *** VALIDATE proc *** [ 0.031296] *** VALIDATE cgroup *** [ 0.032013] *** VALIDATE cgroup2 *** [ 0.033321] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036039] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040397] debug: unmapping init [mem 0xffffffffb9459000-0xffffffffb9460fff] [ 0.042168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043740] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.052179] rcu: Hierarchical SRCU implementation. [ 0.054592] smp: Bringing up secondary CPUs ... [ 0.055595] x86: Booting SMP configuration: [ 0.056029] .... node #0, CPUs: #1 #2 #3 [ 0.060473] smp: Brought up 1 node, 4 CPUs [ 0.062020] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148546] node 0 deferred pages initialised in 81ms [ 0.152412] devtmpfs: initialized [ 0.153240] x86/mm: Memory block size: 128MB [ 0.155694] gcov: version magic: 0x41383552 [ 0.157324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158391] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161489] pinctrl core: initialized pinctrl subsystem [ 0.163219] [ 0.163758] ************************************************************* [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ** ** [ 0.171011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173015] ** ** [ 0.175015] ** This means that this kernel is built to expose internal ** [ 0.177013] ** IOMMU data structures, which may compromise security on ** [ 0.180015] ** your system. ** [ 0.182011] ** ** [ 0.184013] ** If you see this message and you are not debugging the ** [ 0.186013] ** kernel, report this immediately to your vendor! ** [ 0.188022] ** ** [ 0.190014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193015] ************************************************************* [ 0.196653] NET: Registered protocol family 16 [ 0.198858] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208099] cpuidle: using governor menu [ 0.211000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214564] PCI: Using configuration type 1 for base access [ 0.216181] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225173] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227202] cryptd: max_cpu_qlen set to 1000 [ 0.230434] ACPI: Added _OSI(Module Device) [ 0.231013] ACPI: Added _OSI(Processor Device) [ 0.233013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239718] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249553] ACPI: Interpreter enabled [ 0.251067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252015] ACPI: Using IOAPIC for interrupt routing [ 0.254108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265319] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277806] acpiphp: Slot [2] registered [ 0.279218] acpiphp: Slot [3] registered [ 0.283163] acpiphp: Slot [4] registered [ 0.285830] acpiphp: Slot [5] registered [ 0.287172] acpiphp: Slot [6] registered [ 0.290194] acpiphp: Slot [7] registered [ 0.292155] acpiphp: Slot [8] registered [ 0.295176] acpiphp: Slot [9] registered [ 0.298126] acpiphp: Slot [10] registered [ 0.299079] acpiphp: Slot [11] registered [ 0.303164] acpiphp: Slot [12] registered [ 0.305098] acpiphp: Slot [13] registered [ 0.308154] acpiphp: Slot [14] registered [ 0.311079] acpiphp: Slot [15] registered [ 0.313224] acpiphp: Slot [16] registered [ 0.315999] acpiphp: Slot [17] registered [ 0.317089] acpiphp: Slot [18] registered [ 0.320120] acpiphp: Slot [19] registered [ 0.322123] acpiphp: Slot [20] registered [ 0.324124] acpiphp: Slot [21] registered [ 0.326099] acpiphp: Slot [22] registered [ 0.328130] acpiphp: Slot [23] registered [ 0.330158] acpiphp: Slot [24] registered [ 0.331146] acpiphp: Slot [25] registered [ 0.332081] acpiphp: Slot [26] registered [ 0.333076] acpiphp: Slot [27] registered [ 0.334062] acpiphp: Slot [28] registered [ 0.335064] acpiphp: Slot [29] registered [ 0.337127] acpiphp: Slot [30] registered [ 0.340147] acpiphp: Slot [31] registered [ 0.342081] PCI host bridge to bus 0000:00 [ 0.343025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346050] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.349034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.354035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.358034] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.361028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.367224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373565] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.384113] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397489] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.405000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.414019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.416020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.421573] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.425771] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.429052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.431652] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.437994] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.452654] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.460018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.466724] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.476019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.486024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.506026] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.523000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.533023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.540019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.563019] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.574000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.577412] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.583059] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.586462] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.589219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.599129] iommu: Default domain type: Passthrough [ 0.601494] SCSI subsystem initialized [ 0.603182] ACPI: bus type USB registered [ 0.605648] usbcore: registered new interface driver usbfs [ 0.609138] usbcore: registered new interface driver hub [ 0.612164] usbcore: registered new device driver usb [ 0.615213] pps_core: LinuxPPS API ver. 1 registered [ 0.618017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.619103] PTP clock support registered [ 0.629239] EDAC MC: Ver: 3.0.0 [ 0.632264] PCI: Using ACPI for IRQ routing [ 0.633000] NetLabel: Initializing [ 0.634012] NetLabel: domain hash size = 128 [ 0.636012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.638094] NetLabel: unlabeled traffic allowed by default [ 0.641536] vgaarb: loaded [ 0.643309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.645014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.655162] clocksource: Switched to clocksource kvm-clock [ 0.784165] VFS: Disk quotas dquot_6.6.0 [ 0.785880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.788565] *** VALIDATE ramfs *** [ 0.789847] *** VALIDATE hugetlbfs *** [ 0.791554] pnp: PnP ACPI init [ 0.793715] pnp: PnP ACPI: found 6 devices [ 0.811146] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.814465] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.817095] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.819253] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.821691] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.824462] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.827373] NET: Registered protocol family 2 [ 0.830051] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.835522] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.839382] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.844560] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.848983] TCP: Hash tables configured (established 65536 bind 65536) [ 0.852134] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.855702] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858791] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.861788] NET: Registered protocol family 1 [ 0.870225] RPC: Registered named UNIX socket transport module. [ 0.872677] RPC: Registered udp transport module. [ 0.874396] RPC: Registered tcp transport module. [ 0.876428] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.879462] NET: Registered protocol family 44 [ 0.881194] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.883735] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885798] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888177] PCI: CLS 0 bytes, default 64 [ 0.890787] Unpacking initramfs... [ 2.530250] debug: unmapping init [mem 0xffff8e2d3cc64000-0xffff8e2d3ffcffff] [ 2.535932] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.538345] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.541854] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.058457] Initialise system trusted keyrings [ 3.060294] Key type blacklist registered [ 3.075836] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.085738] zbud: loaded [ 3.089040] *** VALIDATE nfs *** [ 3.090441] *** VALIDATE nfs4 *** [ 3.093377] pstore: using deflate compression [ 3.097171] Platform Keyring initialized [ 3.253515] NET: Registered protocol family 38 [ 3.255255] Key type asymmetric registered [ 3.257217] Asymmetric key parser 'x509' registered [ 3.258992] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.262151] io scheduler mq-deadline registered [ 3.263821] io scheduler kyber registered [ 3.265277] io scheduler bfq registered [ 3.267315] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.270229] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.273191] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.276187] ACPI: Power Button [PWRF] [ 3.375349] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.469432] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.570602] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.606018] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.644678] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.649560] Non-volatile memory driver v1.3 [ 3.653242] Linux agpgart interface v0.103 [ 3.697537] virtio_blk virtio1: [vda] 133280 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.700651] vda: detected capacity change from 0 to 68239360 [ 3.722234] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.725215] vdb: detected capacity change from 0 to 1073741824 [ 3.732309] libphy: Fixed MDIO Bus: probed [ 3.741496] usbcore: registered new interface driver usbserial_generic [ 3.746049] usbserial: USB Serial support registered for generic [ 3.748659] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.755654] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.757612] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.761923] mousedev: PS/2 mouse device common for all mice [ 3.767481] rtc_cmos 00:05: RTC can wake from S4 [ 3.773542] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.774921] rtc_cmos 00:05: registered as rtc0 [ 3.780421] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.784330] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.784761] intel_pstate: CPU model not supported [ 3.790783] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.794354] hid: raw HID events driver (C) Jiri Kosina [ 3.796898] usbcore: registered new interface driver usbhid [ 3.799159] usbhid: USB HID core driver [ 3.800694] drop_monitor: Initializing network drop monitor service [ 3.803245] Initializing XFRM netlink socket [ 3.805049] NET: Registered protocol family 10 [ 3.808435] Segment Routing with IPv6 [ 3.810044] NET: Registered protocol family 17 [ 3.812286] mpls_gso: MPLS GSO support [ 3.817317] RAS: Correctable Errors collector initialized. [ 3.819659] AVX version of gcm_enc/dec engaged. [ 3.821495] AES CTR mode by8 optimization enabled [ 3.912216] sched_clock: Marking stable (3912195140, 0)->(4877330009, -965134869) [ 3.916245] registered taskstats version 1 [ 3.918602] Loading compiled-in X.509 certificates [ 3.920719] zswap: loaded using pool lzo/zbud [ 3.954888] Key type big_key registered [ 3.967493] Key type encrypted registered [ 3.968990] ima: No TPM chip found, activating TPM-bypass! [ 3.971228] ima: Allocated hash algorithm: sha1 [ 3.972938] ima: No architecture policies found [ 3.974892] evm: Initialising EVM extended attributes: [ 3.976956] evm: security.selinux [ 3.978469] evm: security.ima [ 3.979831] evm: security.capability [ 3.981523] evm: HMAC attrs: 0x1 [ 3.985243] rtc_cmos 00:05: setting system clock to 2025-11-07 04:17:37 UTC (1762489057) [ 3.997270] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 4.000340] debug: unmapping init [mem 0xffffffffb9182000-0xffffffffb9458fff] [ 4.012086] Write protecting the kernel read-only data: 28672k [ 4.015252] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 4.017803] debug: unmapping init [mem 0xffffffffb8114000-0xffffffffb81fffff] [ 4.058129] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.067517] systemd[1]: Detected virtualization kvm. [ 4.070150] systemd[1]: Detected architecture x86-64. [ 4.072252] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.101363] systemd[1]: No hostname configured. [ 4.104131] systemd[1]: Set hostname to . [ 4.107213] random: systemd: uninitialized urandom read (16 bytes read) [ 4.110172] systemd[1]: Initializing machine ID from random generator. [ 4.264313] random: systemd: uninitialized urandom read (16 bytes read) [ 4.267110] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.274036] random: systemd: uninitialized urandom read (16 bytes read) [ 4.276581] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.280289] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.930366] device-mapper: uevent: version 1.0.3 [ 4.932444] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.672476] virtio_net virtio0 ens2: renamed from eth0 [ 5.704894] random: fast init done [ 5.847165] scsi host0: ata_piix [ 5.875386] scsi host1: ata_piix [ 5.877187] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.879805] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.000558] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.490501] random: crng init done [ 10.492040] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.011499] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.219871] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.593884] SELinux: Disabled at runtime. [ 13.665823] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.676125] systemd[1]: Detected virtualization kvm. [ 13.678279] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.416377] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.420941] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.428741] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.433980] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.438197] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.450303] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.463958] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 14.585808] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 15.150063] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.489946] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.619532] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.658748] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.675875] EDAC sbridge: Ver: 1.1.2 [ 18.956146] Key type dns_resolver registered [ 19.258521] NFS: Registering the id_resolver key type [ 19.260648] Key type id_resolver registered [ 19.261708] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg325-client login: [ 62.775128] libcfs: loading out-of-tree module taints kernel. [ 62.978950] Key type ._llcrypt registered [ 62.981169] Key type .llcrypt registered [ 63.460862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 63.473230] alg: No test for adler32 (adler32-zlib) [ 64.676607] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 65.378905] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 67.112474] Key type lgssc registered [ 68.771785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.550442] hrtimer: interrupt took 3981303 ns [ 158.690783] Lustre: Mounted lustre-client [ 162.247514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 174.857959] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing check_logdir /tmp/testlogs/ [ 177.903548] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing yml_node [ 181.158686] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 182.908411] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 184.291485] Lustre: lustre-OST0000-osc-ffff8e2d89211800: disconnect after 23s idle [ 184.658547] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 185.707500] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Nov 6 23:20:38 EST 2025 [ 195.555630] LustreError: 7910:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e2d89211800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 195.563592] LustreError: 7910:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 195.593086] Lustre: Unmounted lustre-client [ 211.004403] Key type lgssc unregistered [ 211.216608] LNet: 8356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 211.222873] LNetError: 8356:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 212.262352] LNet: Removed LNI 192.168.203.25@tcp [ 212.754633] Key type .llcrypt unregistered [ 212.756108] Key type ._llcrypt unregistered [ 229.647438] Key type ._llcrypt registered [ 229.648965] Key type .llcrypt registered [ 229.794267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 229.805413] alg: No test for adler32 (adler32-zlib) [ 230.738172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 230.798034] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 230.811871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 233.409745] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 233.473970] LNet: 9224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.480436] LNetError: 9224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 233.504591] LNet: Removed LNI 192.168.203.25@tcp [ 233.943989] Key type .llcrypt unregistered [ 233.946658] Key type ._llcrypt unregistered [ 237.535241] Key type ._llcrypt registered [ 237.537230] Key type .llcrypt registered [ 238.395403] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:21:30 (1762489290) [ 238.630959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 238.639389] alg: No test for adler32 (adler32-zlib) [ 239.523785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.575859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.592052] LNet: 9635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 239.610525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.656421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 239.680857] LNet: 9642:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 244.550879] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:21:37 (1762489297) [ 244.599368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.614522] LNet: 10213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 244.630335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.686932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 248.976598] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:21:41 (1762489301) [ 249.045971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.070841] LNet: 10797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 249.098648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.218246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 253.322615] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:21:45 (1762489305) [ 253.407743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.426199] LNet: 11381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 253.441569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.504644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 258.262381] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:21:50 (1762489310) [ 258.379636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.398077] LNet: 11971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.413915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.462641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 258.735684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 263.484979] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:21:55 (1762489315) [ 263.698387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.747639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.814112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 264.161891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 264.533395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 268.540974] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:22:01 (1762489321) [ 268.597787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.611760] LNet: 13152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.617102] LNet: 13152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 268.640612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.694216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 273.071468] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:22:05 (1762489325) [ 273.162293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.189220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.235538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 273.261781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 273.514314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 273.536476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 273.744936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 273.771981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 274.039616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 274.066436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 274.325491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 274.350563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 274.620516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 274.649127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 274.942875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 274.979511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 275.221747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 275.257472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 275.503552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 275.548943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 275.809752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 275.847184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 276.074500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 276.116077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 276.533078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 276.565307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 276.851402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 276.873287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 281.483599] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:22:14 (1762489334) [ 281.574490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.609513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.688637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 281.711362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 286.215135] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:22:18 (1762489338) [ 286.399330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.418105] LNet: 15007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.421366] LNet: 15007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 286.445406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.518606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 286.548665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 291.630472] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:22:24 (1762489344) [ 291.757454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.802959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.864921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 291.891344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 296.505862] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:22:29 (1762489349) [ 296.600212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.637767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.727862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 296.761640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 301.691743] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:22:34 (1762489354) [ 301.766252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.799302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.871812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 301.894420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 306.787237] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:22:39 (1762489359) [ 306.872106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.902408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.981863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 307.006610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 311.944339] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:22:44 (1762489364) [ 311.991520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.021841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.105261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 312.398225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 318.084971] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:22:50 (1762489370) [ 318.306885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.341564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.544486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 318.782788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 325.464411] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:22:57 (1762489377) [ 325.561350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.583467] LNet: 19151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.586331] LNet: 19151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 325.612804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.695179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 325.888051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 330.682266] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:23:03 (1762489383) [ 330.758073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.782420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.192178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 331.428895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 336.201662] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:23:08 (1762489388) [ 336.351316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.383361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.443453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 336.494470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 342.995280] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:23:15 (1762489395) [ 343.086920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.119933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.206806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 343.239295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 348.253900] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:23:20 (1762489400) [ 348.415368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.525430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.575514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 353.719642] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:23:25 (1762489405) [ 353.782695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.823571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.908967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 359.070995] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:23:31 (1762489411) [ 359.309703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.344872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.418349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 359.765854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 364.870692] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:23:37 (1762489417) [ 364.952514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.023768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.124848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 365.176299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 369.909919] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:23:42 (1762489422) [ 370.166187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.215510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.309626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 370.354489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 374.708956] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:23:47 (1762489427) [ 374.965964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.005953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.072401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 375.101638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 380.106958] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:23:52 (1762489432) [ 380.339555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.376291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.457629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 380.502167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 385.465913] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:23:57 (1762489437) [ 385.549269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.606847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.667774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 385.735887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 385.759380] LNetError: 25738:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 385.789680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 385.820910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 390.385972] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:24:02 (1762489442) [ 390.561386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.591416] LNet: 26315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.596182] LNet: 26315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 390.611700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.669592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Eu4uZuCd5O [ 390.730413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Eu4uZuCd5O [ 390.778360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Eu4uZuCd5O [ 395.232613] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:24:07 (1762489447) [ 395.386532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.416775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.456313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 395.479928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 395.513693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 399.773360] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:24:12 (1762489452) [ 399.867341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.896384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.941491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 401.436432] LNet: Message forwarding will be enabled in 2s [ 404.512130] LNet: Message forwarding enabled [ 404.748965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 404.769183] LNet: Message forwarding will stop in 2s to 6s [ 409.092170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 409.419101] LNet: Message forwarding disabled [ 410.478257] LNet: Message forwarding will be enabled in 2s [ 414.560122] LNet: Message forwarding enabled [ 414.968528] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:24:27 (1762489467) [ 415.097643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.907818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.981542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 420.575032] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:24:32 (1762489472) [ 420.647205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.680755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.749944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 420.795089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 420.821924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 420.844142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 420.868837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 420.900848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 420.927122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 420.957115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 420.985178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 421.021194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 421.057436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 421.085984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 421.116906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 421.145698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 421.182882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 421.214873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 421.240421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 421.262635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 421.281699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 421.301900] LNetError: 28807:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 425.878737] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:24:38 (1762489478) [ 425.962166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.002519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.052042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 431.203947] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:24:43 (1762489483) [ 432.527670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg325-server [ 433.824146] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:24:46 (1762489486) [ 433.902111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.937756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.994730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 434.019463] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 435.160481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[26-30]@tcp [ 435.183339] LNet: 30520:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 435.227916] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.28@tcp is being used as a gateway but routing feature is not turned on [ 435.246620] LNetError: 30313:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.28@tcp has gone from up to down [ 436.260040] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 436.272469] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 437.296200] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.27@tcp is being used as a gateway but routing feature is not turned on [ 437.301377] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 439.333655] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.26@tcp is being used as a gateway but routing feature is not turned on [ 439.347075] LNetError: 30313:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 439.657083] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:24:52 (1762489492) [ 440.912260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg325-server [ 442.125819] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:24:54 (1762489494) [ 442.183898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.196424] LNetError: 31448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 442.217812] LNet: Removed LNI 192.168.203.25@tcp [ 442.261636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.327395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 442.351785] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 442.834408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[26-31/2]@tcp [ 442.849195] LNet: 31569:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 442.856895] LNet: 31569:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 442.871878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[26-31/2]@tcp [ 442.878041] LNetError: 31458:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.26@tcp has gone from up to down [ 442.902148] LNetError: 31458:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 447.650178] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:25:00 (1762489500) [ 447.907947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.935074] LNetError: 32160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 447.955940] LNet: Removed LNI 192.168.203.25@tcp [ 447.971473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.053759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 448.102069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.123171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.187379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.221976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.247647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.288497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 448.343078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.373720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.407916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.459383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.481548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.514184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 448.565321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.595980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.672288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.725703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.756512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.787669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 448.834928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.859602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.912888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.965592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.997418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.032071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 449.085338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.115456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.173889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.245338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.276523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.332325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 449.384464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.411480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.453185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 453.731487] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:25:06 (1762489506) [ 455.123789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg325-server [ 456.504407] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:25:08 (1762489508) [ 457.857844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg325-server [ 459.049783] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:25:11 (1762489511) [ 459.873327] Key type .llcrypt unregistered [ 459.875308] Key type ._llcrypt unregistered [ 463.686741] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 463.762213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 463.770798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 463.775335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 466.251841] Key type ._llcrypt registered [ 466.253137] Key type .llcrypt registered [ 466.434100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 466.443741] alg: No test for adler32 (adler32-zlib) [ 467.310514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.360783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 467.390234] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 467.943494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 467.957799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 469.452399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 469.471877] LNet: Removed LNI 10.1.2.3@tcp [ 469.596982] LNet: 34271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.602976] LNetError: 34271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.614242] LNet: Removed LNI 192.168.203.25@tcp [ 470.125642] Key type .llcrypt unregistered [ 470.127568] Key type ._llcrypt unregistered [ 477.799184] Key type ._llcrypt registered [ 477.801014] Key type .llcrypt registered [ 478.670518] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:25:31 (1762489531) [ 479.115604] Key type .llcrypt unregistered [ 479.117447] Key type ._llcrypt unregistered [ 483.433786] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 483.595247] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 483.609285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 483.611914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 486.099429] Key type ._llcrypt registered [ 486.101245] Key type .llcrypt registered [ 486.235560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 486.250359] alg: No test for adler32 (adler32-zlib) [ 487.127657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.170784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 487.194630] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 487.747691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 487.776804] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 489.868805] LNet: 35749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.872768] LNetError: 35749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.879219] LNet: Removed LNI 192.168.203.25@tcp [ 489.887284] LNet: Removed LNI 10.1.2.3@tcp [ 490.361644] Key type .llcrypt unregistered [ 490.363111] Key type ._llcrypt unregistered [ 498.971829] Key type ._llcrypt registered [ 498.973719] Key type .llcrypt registered [ 500.055819] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:25:52 (1762489552) [ 500.488364] Key type .llcrypt unregistered [ 500.490071] Key type ._llcrypt unregistered [ 505.148471] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 505.309341] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 505.313426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 505.315394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 507.804321] Key type ._llcrypt registered [ 507.805950] Key type .llcrypt registered [ 507.948709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 507.975104] alg: No test for adler32 (adler32-zlib) [ 508.910992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.978985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 509.003301] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 510.560575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 510.580509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 512.335994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 512.351156] LNet: Removed LNI 10.1.2.3@tcp [ 512.373332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 512.400795] LNet: Removed LNI 10.1.2.31@tcp [ 512.425157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 512.463553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 513.053799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 513.075569] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 513.922202] LNet: 37642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.927422] LNetError: 37642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 513.933695] LNet: Removed LNI 10.1.2.3@tcp [ 513.945274] LNet: Removed LNI 10.1.2.31@tcp [ 514.530611] Key type .llcrypt unregistered [ 514.533589] Key type ._llcrypt unregistered [ 523.482517] Key type ._llcrypt registered [ 523.483823] Key type .llcrypt registered [ 524.396568] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:26:16 (1762489576) [ 524.692391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 524.707146] alg: No test for adler32 (adler32-zlib) [ 525.591865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.630097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.651636] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 526.151963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.165222] LNet: 38585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.168299] LNetError: 38585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 526.183782] LNet: Removed LNI 192.168.203.25@tcp [ 526.195818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.223988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.25@tcp [ 526.241154] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 526.752210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.25@tcp [ 526.777416] LNet: Removed LNI 192.168.203.25@tcp [ 526.836244] LNet: 38717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.283516] Key type .llcrypt unregistered [ 527.285876] Key type ._llcrypt unregistered [ 534.610431] Key type ._llcrypt registered [ 534.612053] Key type .llcrypt registered [ 535.618529] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:26:28 (1762489588) [ 537.268463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg325-server [ 538.667653] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:26:31 (1762489591) [ 539.120372] Key type .llcrypt unregistered [ 539.131292] Key type ._llcrypt unregistered [ 543.116019] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 543.161334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 543.168744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 543.175534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 545.748505] Key type ._llcrypt registered [ 545.754109] Key type .llcrypt registered [ 545.955774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 545.973218] alg: No test for adler32 (adler32-zlib) [ 546.856546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.906751] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 547.426483] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 549.209185] LNet: 40535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.216689] LNetError: 40535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.223877] LNet: Removed LNI 192.168.203.25@tcp1 [ 549.234048] LNet: Removed LNI 10.1.2.3@tcp2 [ 549.832555] Key type .llcrypt unregistered [ 549.836238] Key type ._llcrypt unregistered [ 558.203856] Key type ._llcrypt registered [ 558.205859] Key type .llcrypt registered [ 559.333818] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:26:51 (1762489611) [ 559.753962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.772328] alg: No test for adler32 (adler32-zlib) [ 560.639702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.693865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 560.732673] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 561.387821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.422253] LNet: 41492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.429408] LNetError: 41492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 561.445173] LNet: Removed LNI 192.168.203.25@tcp [ 561.464182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.583636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 561.599682] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 567.620276] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:26:59 (1762489619) [ 567.809152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.826480] LNet: 42088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.833338] LNetError: 42088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.849345] LNet: Removed LNI 192.168.203.25@tcp [ 567.870915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.956923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 567.993236] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 568.655945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.25 [ 568.667414] LNet: Removed LNI 192.168.203.25@tcp [ 573.707852] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:27:06 (1762489626) [ 573.916098] LNet: 42788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.922820] LNetError: 42788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.935739] LNet: Removed LNI 192.168.203.25@tcp [ 574.507690] Key type .llcrypt unregistered [ 574.509279] Key type ._llcrypt unregistered [ 578.905893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 579.081157] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 579.083881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 579.088277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 581.697651] Key type ._llcrypt registered [ 581.699443] Key type .llcrypt registered [ 581.849825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 581.872691] alg: No test for adler32 (adler32-zlib) [ 583.136276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.930285] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:27:21 (1762489641) [ 589.018816] LNet: 43679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.025460] LNetError: 43679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 589.041523] LNet: Removed LNI 10.1.2.3@tcp [ 589.507334] Key type .llcrypt unregistered [ 589.509056] Key type ._llcrypt unregistered [ 593.872239] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 594.056890] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 594.063892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 594.068106] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 596.648490] Key type ._llcrypt registered [ 596.650218] Key type .llcrypt registered [ 596.813108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 596.823211] alg: No test for adler32 (adler32-zlib) [ 597.753857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 602.413043] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:27:34 (1762489654) [ 602.569814] LNet: 44575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.579604] LNetError: 44575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 602.589790] LNet: Removed LNI 10.1.2.3@tcp [ 603.097668] Key type .llcrypt unregistered [ 603.099284] Key type ._llcrypt unregistered [ 607.305385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 607.472687] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 607.475605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 607.486147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 610.030083] Key type ._llcrypt registered [ 610.032876] Key type .llcrypt registered [ 610.149400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 610.161961] alg: No test for adler32 (adler32-zlib) [ 611.073931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 615.571537] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:27:48 (1762489668) [ 615.655910] LNet: 45454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.662397] LNetError: 45454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 615.692439] LNet: Removed LNI 10.1.2.3@tcp [ 616.224079] Key type .llcrypt unregistered [ 616.225836] Key type ._llcrypt unregistered [ 619.895270] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 620.046809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 620.058771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 620.072808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 622.609542] Key type ._llcrypt registered [ 622.611549] Key type .llcrypt registered [ 622.734274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 622.746197] alg: No test for adler32 (adler32-zlib) [ 623.636944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.703422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 630.585400] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:28:02 (1762489682) [ 630.927652] LNet: 46684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.930915] LNetError: 46684:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 630.946539] LNet: Removed LNI 10.1.2.3@tcp [ 631.510115] Key type .llcrypt unregistered [ 631.511892] Key type ._llcrypt unregistered [ 635.969698] Key type ._llcrypt registered [ 635.977168] Key type .llcrypt registered [ 636.146700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 636.157552] alg: No test for adler32 (adler32-zlib) [ 637.087485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 637.108824] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 643.947549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 644.546913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_interrupt -r 1 [ 644.579658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_interrupt -r 1 [ 644.600721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 644.618193] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 644.856378] LNet: 47108:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 644.922607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 644.950957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 644.980723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 645.394291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_dropped -r 1 [ 645.411153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_dropped -r 1 [ 645.463307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 645.485880] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 645.802347] LNet: 47205:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 645.878483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 645.901439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 645.929208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 646.530045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_aborted -r 1 [ 646.584457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_aborted -r 1 [ 646.627877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 646.649890] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 646.983231] LNet: 47302:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 647.079894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 647.135346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 647.177865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 647.786820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_no_route -r 1 [ 647.837968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_no_route -r 1 [ 647.885794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 648.280483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 648.322403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 648.361657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 649.030654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_timeout -r 1 [ 649.084547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_timeout -r 1 [ 649.147359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 649.180380] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 649.185443] LNet: Skipped 1 previous similar message [ 649.540109] LNet: 47497:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 649.547916] LNet: 47497:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 649.659900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 649.721732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 649.764089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 650.370182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_error -r 1 [ 650.435163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_error -r 1 [ 650.476627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 650.913771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 650.991157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 651.032593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 651.351630] LNet: 47630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.363462] LNetError: 47630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.386493] LNet: Removed LNI 192.168.203.25@tcp [ 651.872602] Key type .llcrypt unregistered [ 651.874050] Key type ._llcrypt unregistered [ 665.417062] Key type ._llcrypt registered [ 665.423111] Key type .llcrypt registered [ 666.510959] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:28:38 (1762489718) [ 667.141414] Key type .llcrypt unregistered [ 667.144388] Key type ._llcrypt unregistered [ 670.918834] Key type ._llcrypt registered [ 670.920564] Key type .llcrypt registered [ 671.033899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.047470] alg: No test for adler32 (adler32-zlib) [ 671.900487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 671.918495] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 676.933094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 684.930680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.964759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 684.993549] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 686.013452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_interrupt -r 1 [ 686.057954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 1 [ 686.086666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_interrupt -r 1 [ 686.123866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 1 [ 686.148279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_interrupt -r 1 [ 686.195296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 1 [ 686.224577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_interrupt -r 1 [ 686.253454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 1 [ 686.307481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 686.337242] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 686.345567] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 686.594620] LNet: 49079:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 686.660971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.696136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 686.734370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 687.228380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_dropped -r 1 [ 687.252359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 1 [ 687.287817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_dropped -r 1 [ 687.336319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 1 [ 687.394027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_dropped -r 1 [ 687.427534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 1 [ 687.461422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_dropped -r 1 [ 687.484311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 1 [ 687.517681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 687.533888] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 687.540087] LNet: Skipped 2 previous similar messages [ 687.804222] LNet: 49188:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 687.883156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 687.921313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 687.957809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 688.397545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_aborted -r 1 [ 688.426823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 1 [ 688.445815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_aborted -r 1 [ 688.496760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 1 [ 688.528956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_aborted -r 1 [ 688.557396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 1 [ 688.587479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_aborted -r 1 [ 688.624604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 1 [ 688.653854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 688.670687] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 688.674791] LNet: Skipped 2 previous similar messages [ 688.908804] LNet: 49297:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 688.978541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 689.009705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 689.046216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 689.528890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_no_route -r 1 [ 689.563657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 1 [ 689.601864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_no_route -r 1 [ 689.634949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 1 [ 689.658575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_no_route -r 1 [ 689.681416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 1 [ 689.714891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_no_route -r 1 [ 689.746021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 1 [ 689.773636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 690.086968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 690.111377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 690.136575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 690.496375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_timeout -r 1 [ 690.532262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 1 [ 690.556372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_timeout -r 1 [ 690.600692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 1 [ 690.629414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_timeout -r 1 [ 690.702424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 1 [ 690.774055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_timeout -r 1 [ 690.797539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 1 [ 690.823724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 690.847168] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 690.852605] LNet: Skipped 5 previous similar messages [ 691.143352] LNet: 49515:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 691.147882] LNet: 49515:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 691.214789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 691.245380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 691.269914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 691.676278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_error -r 1 [ 691.713470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_error -r 1 [ 691.780449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_error -r 1 [ 691.822511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_error -r 1 [ 691.856895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_error -r 1 [ 691.898375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_error -r 1 [ 691.949576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_error -r 1 [ 691.980761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_error -r 1 [ 692.010918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 692.384274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 692.418758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 692.457875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 695.090441] LNet: 49695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.095852] LNetError: 49695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.096208] LNet: Removed LNI 192.168.203.25@tcp [ 696.100631] LNetError: 49695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.110827] LNet: Removed LNI 192.168.203.25@tcp1 [ 696.510674] Key type .llcrypt unregistered [ 696.512351] Key type ._llcrypt unregistered [ 707.918135] Key type ._llcrypt registered [ 707.919849] Key type .llcrypt registered [ 709.263588] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:29:21 (1762489761) [ 709.788538] Key type .llcrypt unregistered [ 709.790218] Key type ._llcrypt unregistered [ 713.211380] Key type ._llcrypt registered [ 713.217638] Key type .llcrypt registered [ 713.342208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.351211] alg: No test for adler32 (adler32-zlib) [ 714.209770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 714.226845] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 718.862580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 719.343679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_dropped -r 1 [ 719.368350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_dropped -r 1 [ 719.403850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 719.417579] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 719.664054] LNet: 50928:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 719.722790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 719.754428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 719.779789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 720.123741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_error -r 1 [ 720.152343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_error -r 1 [ 720.194157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 720.211422] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 720.458139] LNet: 51026:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 720.503839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 720.524544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 720.556321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 720.935269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_timeout -r 1 [ 720.958257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_timeout -r 1 [ 720.983078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 721.216129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 721.242256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 721.270545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 721.437815] LNet: 51159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.446342] LNetError: 51159:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.460721] LNet: Removed LNI 192.168.203.25@tcp [ 721.716657] Key type .llcrypt unregistered [ 721.722596] Key type ._llcrypt unregistered [ 731.207031] Key type ._llcrypt registered [ 731.212508] Key type .llcrypt registered [ 732.276683] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:29:44 (1762489784) [ 732.911751] Key type .llcrypt unregistered [ 732.913593] Key type ._llcrypt unregistered [ 736.689681] Key type ._llcrypt registered [ 736.691677] Key type .llcrypt registered [ 736.838694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 736.859218] alg: No test for adler32 (adler32-zlib) [ 737.693652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 737.712779] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 742.101586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 746.900960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.935716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 746.957712] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 747.825287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_dropped -r 1 [ 747.847579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 1 [ 747.873671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_dropped -r 1 [ 747.892589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 1 [ 747.912295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_dropped -r 1 [ 747.939514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 1 [ 747.964615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_dropped -r 1 [ 747.994137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 1 [ 748.012605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 748.026374] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 748.031344] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 748.210949] LNet: 52604:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 748.250116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 748.285529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 748.309228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 748.332346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 748.654708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_error -r 1 [ 748.704592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_error -r 1 [ 748.743554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_error -r 1 [ 748.767959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_error -r 1 [ 748.805700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_error -r 1 [ 748.831220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_error -r 1 [ 748.861391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_error -r 1 [ 748.879661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_error -r 1 [ 748.900578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 748.916793] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 748.920054] LNet: Skipped 2 previous similar messages [ 749.112797] LNet: 52716:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 749.167901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.186894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 749.217493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 749.244082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.604323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_timeout -r 1 [ 749.631917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_timeout -r 1 [ 749.651296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_timeout -r 1 [ 749.674791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_timeout -r 1 [ 749.706912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_timeout -r 1 [ 749.743851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_timeout -r 1 [ 749.764565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_timeout -r 1 [ 749.792261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_timeout -r 1 [ 749.816505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.125@tcp [ 750.072973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.102361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 750.128989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 750.156524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 753.357155] LNet: 52900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.365753] LNetError: 52900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.373749] LNet: Removed LNI 192.168.203.25@tcp [ 753.383915] LNet: Removed LNI 192.168.203.25@tcp1 [ 753.696170] Key type .llcrypt unregistered [ 753.705173] Key type ._llcrypt unregistered [ 764.295368] Key type ._llcrypt registered [ 764.296886] Key type .llcrypt registered [ 765.203510] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:30:17 (1762489817) [ 765.555271] Key type .llcrypt unregistered [ 765.556964] Key type ._llcrypt unregistered [ 769.423623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.518854] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.521262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.523775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 772.036664] Key type ._llcrypt registered [ 772.038307] Key type .llcrypt registered [ 772.217021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.230606] alg: No test for adler32 (adler32-zlib) [ 773.115986] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 773.146892] LNet: 53984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.151369] LNetError: 53984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.168526] LNet: Removed LNI 192.168.203.25@tcp [ 773.820400] Key type .llcrypt unregistered [ 773.821919] Key type ._llcrypt unregistered [ 778.305596] Key type ._llcrypt registered [ 778.310701] Key type .llcrypt registered [ 778.438137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.445805] alg: No test for adler32 (adler32-zlib) [ 779.340641] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 779.343545] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 779.368112] LNet: 54252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.373686] LNetError: 54252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.376944] LNet: Removed LNI 192.168.203.25@tcp [ 779.390397] LNet: Removed LNI 10.1.2.3@tcp1 [ 779.943461] Key type .llcrypt unregistered [ 779.945241] Key type ._llcrypt unregistered [ 784.121947] Key type ._llcrypt registered [ 784.123294] Key type .llcrypt registered [ 784.260852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.274660] alg: No test for adler32 (adler32-zlib) [ 785.163102] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 785.166582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.204725] LNet: 54520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.211122] LNetError: 54520:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.215327] LNet: Removed LNI 192.168.203.25@tcp [ 785.224592] LNet: Removed LNI 10.1.2.3@tcp [ 785.808526] Key type .llcrypt unregistered [ 785.810225] Key type ._llcrypt unregistered [ 790.209947] Key type ._llcrypt registered [ 790.211859] Key type .llcrypt registered [ 790.356068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.370195] alg: No test for adler32 (adler32-zlib) [ 791.245928] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 791.248727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.296467] LNet: 54790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.300337] LNetError: 54790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.303965] LNet: Removed LNI 192.168.203.25@tcp [ 791.317314] LNet: Removed LNI 10.1.2.3@tcp [ 791.736787] Key type .llcrypt unregistered [ 791.738356] Key type ._llcrypt unregistered [ 795.117716] Key type ._llcrypt registered [ 795.119482] Key type .llcrypt registered [ 795.262989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.276680] alg: No test for adler32 (adler32-zlib) [ 796.180740] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 796.182676] LNetError: 55049:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 796.186796] LNetError: Error -100 starting up LNI tcp [ 796.200935] LNet: Removed LNI 192.168.203.25@tcp [ 796.723526] Key type .llcrypt unregistered [ 796.725053] Key type ._llcrypt unregistered [ 803.149831] Key type ._llcrypt registered [ 803.151374] Key type .llcrypt registered [ 803.968238] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:30:56 (1762489856) [ 804.492560] Key type .llcrypt unregistered [ 804.494522] Key type ._llcrypt unregistered [ 808.063654] Key type ._llcrypt registered [ 808.066798] Key type .llcrypt registered [ 808.205780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.216435] alg: No test for adler32 (adler32-zlib) [ 809.116426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 809.146917] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 813.846767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 814.322502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e network_timeout -r 1 [ 814.346840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e network_timeout -r 1 [ 814.373545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 814.386799] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 814.577531] LNet: 56288:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 814.621822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 814.650947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 814.827253] LNet: 56322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.831893] LNetError: 56322:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.845140] LNet: Removed LNI 192.168.203.25@tcp [ 815.118020] Key type .llcrypt unregistered [ 815.119707] Key type ._llcrypt unregistered [ 824.950928] Key type ._llcrypt registered [ 824.953950] Key type .llcrypt registered [ 825.145390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.158459] alg: No test for adler32 (adler32-zlib) [ 826.026158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 826.096474] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 831.157877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 836.100880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.143266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 836.157098] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 837.318035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e network_timeout -r 1 [ 837.362530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e network_timeout -r 1 [ 837.407895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e network_timeout -r 1 [ 837.473240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e network_timeout -r 1 [ 837.521364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e network_timeout -r 1 [ 837.577372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e network_timeout -r 1 [ 837.633983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e network_timeout -r 1 [ 837.687485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e network_timeout -r 1 [ 837.725659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 837.752202] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 838.108212] LNet: 57183:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 838.179290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.222886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 841.787923] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 842.010152] LNet: 57252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.018394] LNetError: 57252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.040157] LNet: Removed LNI 192.168.203.25@tcp [ 843.041916] LNetError: 57252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.064664] LNet: Removed LNI 192.168.203.25@tcp1 [ 843.375599] Key type .llcrypt unregistered [ 843.378609] Key type ._llcrypt unregistered [ 854.641715] Key type ._llcrypt registered [ 854.643474] Key type .llcrypt registered [ 855.768810] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:31:48 (1762489908) [ 856.079522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.101716] alg: No test for adler32 (adler32-zlib) [ 856.983130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.024600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.037131] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 857.533763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 857.546106] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 858.142926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 858.204631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 858.232901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 858.260025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 858.300693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 858.371230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 858.396697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.25@tcp [ 858.411272] LNetError: 58342:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.203.25@tcp to recovery [ 859.040320] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 859.042919] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 861.088902] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 861.732098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 865.184326] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 866.137488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 866.159978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 866.197180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.210811] LNet: 58459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.214518] LNetError: 58459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.218981] LNet: Removed LNI 192.168.203.25@tcp [ 866.233185] LNet: Removed LNI 192.168.203.25@tcp1 [ 866.249609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.285604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.312929] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 866.828441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 866.843441] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 867.424883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 867.456767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 867.508899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 867.548699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 867.580142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 867.617127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 867.636347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 867.663380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.25@tcp [ 867.678193] LNetError: 58726:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.203.25@tcp to recovery [ 868.320946] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 870.981446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 874.465273] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 874.469758] LNet: Skipped 1 previous similar message [ 879.622471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 879.651559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 879.674688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 884.635083] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:32:16 (1762489936) [ 884.846032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.866567] LNet: 59454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.872825] LNetError: 59454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.881155] LNetError: 59454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.889544] LNet: Removed LNI 192.168.203.25@tcp [ 884.906710] LNet: Removed LNI 192.168.203.25@tcp1 [ 884.927628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.981896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 885.003520] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 885.616368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 885.630359] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 886.241353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 886.284677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 886.305822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 886.343858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 886.362613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 886.410190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 886.435326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.25@tcp --health 0 [ 886.494937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 887.008333] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.25@tcp [ 887.008531] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 887.023496] LNet: Skipped 1 previous similar message [ 901.720853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 901.845141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 901.879200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.920259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.25@tcp --health 500 [ 902.013472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 908.532158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 908.603949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.636927] LNet: 59940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.648022] LNetError: 59940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.651089] LNetError: 59940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 908.657648] LNet: Removed LNI 192.168.203.25@tcp [ 908.669598] LNet: Removed LNI 192.168.203.25@tcp1 [ 908.700353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.806336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 908.856561] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 909.515509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 909.539198] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 910.326420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 910.363259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.428166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 910.448300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 910.473635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 910.501752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 910.529878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 910.560713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.25@tcp --health 0 [ 911.840623] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 911.846613] LNet: Skipped 3 previous similar messages [ 912.850293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 921.687468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 921.734119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 921.770120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.351315] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:32:58 (1762489978) [ 926.654825] LNet: 60985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.658538] LNetError: 60985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.664324] LNetError: 60985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 926.668725] LNet: Removed LNI 192.168.203.25@tcp [ 926.688760] LNet: Removed LNI 192.168.203.25@tcp1 [ 927.082761] Key type .llcrypt unregistered [ 927.084558] Key type ._llcrypt unregistered [ 930.599730] Key type ._llcrypt registered [ 930.605200] Key type .llcrypt registered [ 930.761742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.779539] alg: No test for adler32 (adler32-zlib) [ 931.646802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 931.669641] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 937.706318] Lustre: DEBUG MARKER: Initial discovery [ 937.768883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.125@tcp [ 939.336986] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 939.510158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 939.537657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.125@tcp [ 939.553501] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 939.568738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 939.593867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.125@tcp -r 1 -m GET -l 3 [ 939.623798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.125@tcp -r 1 -m GET -e local_error [ 939.668125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.125@tcp -r 1 -m PUT -l 6 [ 940.634109] Lustre: DEBUG MARKER: Force oleg325-client.virtnet to discover 192.168.203.125@tcp (in background) [ 940.835587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.125@tcp [ 941.778842] Lustre: DEBUG MARKER: Force oleg325-server to discover 192.168.203.25@tcp [ 943.635989] Lustre: DEBUG MARKER: Wait for 61743 [ 943.906281] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 944.968336] Lustre: DEBUG MARKER: Finished wait on 61743 [ 945.026456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 945.060333] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 945.072834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 945.264585] LNet: 62190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.276260] LNetError: 62190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.345763] LNet: Removed LNI 192.168.203.25@tcp [ 946.657479] Key type .llcrypt unregistered [ 946.662230] Key type ._llcrypt unregistered [ 958.449295] Key type ._llcrypt registered [ 958.454047] Key type .llcrypt registered [ 959.471514] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:33:31 (1762490011) [ 959.875642] Key type .llcrypt unregistered [ 959.878736] Key type ._llcrypt unregistered [ 963.846227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 964.004367] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 964.006729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 964.008943] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 966.347268] Key type ._llcrypt registered [ 966.348520] Key type .llcrypt registered [ 966.495796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.504933] alg: No test for adler32 (adler32-zlib) [ 967.389940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.432752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.454951] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 967.938478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 967.955236] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 974.650536] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:33:46 (1762490026) [ 974.742514] LNet: 64251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.750601] LNetError: 64251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.755717] LNet: Removed LNI 192.168.203.25@tcp [ 974.765259] LNet: Removed LNI 10.1.2.3@tcp [ 975.416886] Key type .llcrypt unregistered [ 975.418694] Key type ._llcrypt unregistered [ 979.871738] Key type ._llcrypt registered [ 979.873496] Key type .llcrypt registered [ 980.027235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.037798] alg: No test for adler32 (adler32-zlib) [ 980.898573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.944466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 980.961299] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 981.477730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 981.493110] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.154274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.25@tcp [ 983.200142] LNetError: Unexpected error -22 connecting to 192.168.203.25@tcp at host 192.168.203.25:988 [ 986.747470] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:33:59 (1762490039) [ 986.842415] LNet: 65371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.851852] LNetError: 65371:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.858232] LNet: Removed LNI 192.168.203.25@tcp [ 986.871946] LNet: Removed LNI 10.1.2.3@tcp [ 987.552381] Key type .llcrypt unregistered [ 987.553989] Key type ._llcrypt unregistered [ 991.651256] Key type ._llcrypt registered [ 991.664291] Key type .llcrypt registered [ 991.862562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.875551] alg: No test for adler32 (adler32-zlib) [ 992.771558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.826914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 992.856830] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 993.472400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 993.483975] LNet: Added LNI 192.168.203.25@tcp2 [8/256/0/180] [ 994.208885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.25@tcp1 --nid 192.168.203.25@tcp2 [ 997.904608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp1 192.168.203.25@tcp2 [ 997.932226] LNetError: 66205:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.25@tcp1 to 192.168.203.25@tcp2 - Route table may be misconfigured [ 997.950599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp2 192.168.203.25@tcp1 [ 1002.690850] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:34:15 (1762490055) [ 1002.917566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.943054] LNet: 66778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.948973] LNetError: 66778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.955612] LNet: Removed LNI 192.168.203.25@tcp1 [ 1004.012234] LNet: Removed LNI 192.168.203.25@tcp2 [ 1004.029434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.067557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1004.096048] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1004.580903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1004.599053] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1005.147733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1005.178754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.25@tcp [ 1005.207369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp1 192.168.203.25@tcp1 [ 1005.242164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.25@tcp -d 192.168.203.25@tcp -e network_timeout [ 1005.271624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -e network_timeout [ 1005.321489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -e network_timeout [ 1005.350622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -e network_timeout [ 1005.372983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1005.391566] LNet: There was an unexpected network error while writing to 192.168.203.25: rc = -22 [ 1005.407933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1005.495705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1005.557367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1006.112224] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 1009.882696] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:34:22 (1762490062) [ 1009.966461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.979195] LNet: 67629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.982687] LNetError: 67629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.986037] LNetError: 67629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1009.989666] LNet: Removed LNI 192.168.203.25@tcp [ 1009.999838] LNet: Removed LNI 192.168.203.25@tcp1 [ 1010.013071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.064121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1010.487321] Key type .llcrypt unregistered [ 1010.489421] Key type ._llcrypt unregistered [ 1019.065794] Key type ._llcrypt registered [ 1019.067345] Key type .llcrypt registered [ 1020.335958] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:34:32 (1762490072) [ 1020.805466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.827806] alg: No test for adler32 (adler32-zlib) [ 1021.763162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.125442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1024.990223] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:34:37 (1762490077) [ 1025.108109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.137722] LNet: 68760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.176358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.248950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1025.294310] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1025.927510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1025.949803] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1026.734055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1026.764273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp1 [ 1026.825752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp1 [ 1032.508815] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:34:44 (1762490084) [ 1034.033024] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg325-server [ 1035.471175] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:34:47 (1762490087) [ 1036.863978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg325-server [ 1038.238499] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:34:50 (1762490090) [ 1039.696691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg325-server [ 1041.076330] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:34:53 (1762490093) [ 1042.479715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg325-server [ 1044.148835] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:34:56 (1762490096) [ 1045.654293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg325-server [ 1047.222226] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:34:59 (1762490099) [ 1048.643553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg325-server [ 1050.013984] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:35:02 (1762490102) [ 1051.434712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg325-server [ 1052.830979] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:35:05 (1762490105) [ 1054.280810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg325-server [ 1055.345171] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:35:07 (1762490107) [ 1056.709311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg325-server [ 1058.016412] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:35:10 (1762490110) [ 1058.120994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.200508] LNet: 72629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.225132] LNetError: 72629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.238888] LNet: Removed LNI 192.168.203.25@tcp [ 1059.304199] LNet: Removed LNI 192.168.203.25@tcp1 [ 1059.331282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.384518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1059.419755] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1059.968173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1060.052322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1060.212981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.237551] LNet: 72778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.243817] LNetError: 72778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.249170] LNetError: 72778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1060.278720] LNet: Removed LNI 192.168.203.25@tcp [ 1060.313268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.362127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1060.400470] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1061.101139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1061.296169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.316871] LNetError: 72921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.329599] LNet: Removed LNI 192.168.203.25@tcp [ 1061.357745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.424052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1061.473975] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1062.189445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1062.251432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1063.556735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.578670] LNet: 73075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.582991] LNet: 73075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1063.594996] LNetError: 73075:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.608898] LNet: Removed LNI 192.168.203.25@tcp [ 1063.637853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.712182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1063.757366] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1064.613761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1064.912516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.023853] LNet: Removed LNI 192.168.203.25@tcp [ 1066.092084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.145299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1066.175407] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1066.937377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1067.038456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1067.212726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1068.263099] LNet: Removed LNI 192.168.203.25@tcp [ 1068.291541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.386470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1068.454193] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1069.263914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1069.475672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1069.492922] LNet: 73504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.496393] LNet: 73504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1069.501274] LNetError: 73504:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.505813] LNetError: 73504:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1069.520392] LNet: Removed LNI 192.168.203.25@tcp [ 1069.541824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.601459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1069.630547] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1070.456957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1070.542953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1070.823746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.914715] LNet: Removed LNI 192.168.203.25@tcp [ 1071.948482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.009864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1072.050232] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1072.730853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1074.171814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1075.240501] LNet: Removed LNI 192.168.203.25@tcp [ 1075.280899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.354926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1075.389696] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1076.071196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1076.149346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1077.582558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.604889] LNet: 73953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.609834] LNet: 73953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1077.621956] LNetError: 73953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.625927] LNetError: 73953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1077.637731] LNet: Removed LNI 192.168.203.25@tcp [ 1077.657438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.712780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1077.736513] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1078.409655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1078.699748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1079.784896] LNet: Removed LNI 192.168.203.25@tcp [ 1079.811550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.870989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1079.895395] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1080.432490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1080.483597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1081.939662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.987509] LNet: Removed LNI 192.168.203.25@tcp [ 1083.016266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.119974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1083.156204] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1083.780164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1083.995515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.102137] LNet: Removed LNI 192.168.203.25@tcp [ 1085.130412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.172930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.200891] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1085.677151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1085.730102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.759988] LNet: Removed LNI 192.168.203.25@tcp [ 1085.777099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.810449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.828534] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1086.356199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1091.007564] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:35:43 (1762490143) [ 1091.089368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1091.112344] LNet: Removed LNI 192.168.203.25@tcp [ 1091.137894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.190785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1091.208220] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 1091.997927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1092.024273] LNet: Removed LNI 192.168.203.25@tcp231 [ 1092.042122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.107790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1092.124490] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 1092.132741] LNet: 75364:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.433395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1092.462564] LNet: Removed LNI 192.168.203.25@tcp231 [ 1092.475085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1092.485977] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 1093.263956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.291293] LNet: Removed LNI 192.168.203.25@tcp231 [ 1093.306942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.365670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1093.393932] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/0] [ 1093.398586] LNet: 75513:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1093.681897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1093.709252] LNet: Removed LNI 192.168.203.25@tcp231 [ 1093.726521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1093.746382] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/0] [ 1094.535378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1094.552226] LNet: 75653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.556381] LNet: 75653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1094.563829] LNetError: 75653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.568034] LNetError: 75653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1094.583378] LNet: Removed LNI 192.168.203.25@tcp231 [ 1094.606636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.670043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1094.699188] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/60] [ 1094.703762] LNet: 75661:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.994670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1095.017231] LNet: Removed LNI 192.168.203.25@tcp231 [ 1095.029816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1095.049896] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/60] [ 1095.827393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.852678] LNet: Removed LNI 192.168.203.25@tcp231 [ 1095.869414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.908357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1095.922423] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 1096.153632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1096.190351] LNet: Removed LNI 192.168.203.25@tcp231 [ 1096.200270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1096.223247] LNet: Added LNI 192.168.203.25@tcp231 [8/256/0/180] [ 1100.885602] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:35:53 (1762490153) [ 1100.944572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.973301] LNet: Removed LNI 192.168.203.25@tcp231 [ 1100.996899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.052299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1101.076507] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 1101.534304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1101.559214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1101.580261] LNet: Removed LNI 192.168.203.25@tcp232 [ 1101.592078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1101.609942] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 1102.126844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1102.160377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1102.190514] LNet: Removed LNI 192.168.203.25@tcp232 [ 1102.200870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1102.235705] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 1102.730095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1102.757996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1102.780754] LNet: Removed LNI 192.168.203.25@tcp232 [ 1102.796423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1102.816311] LNet: Added LNI 192.168.203.25@tcp232 [8/256/0/180] [ 1103.270683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1103.291366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1103.312708] LNet: Removed LNI 192.168.203.25@tcp232 [ 1103.323524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1107.186596] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:35:59 (1762490159) [ 1107.959311] Key type .llcrypt unregistered [ 1107.961100] Key type ._llcrypt unregistered [ 1111.813434] Key type ._llcrypt registered [ 1111.815937] Key type .llcrypt registered [ 1111.986384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.003434] alg: No test for adler32 (adler32-zlib) [ 1112.867214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1112.891341] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1118.753978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1124.673169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1124.719424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1124.757773] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1125.971915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_interrupt -r 2 [ 1125.999623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 2 [ 1126.045232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_interrupt -r 2 [ 1126.078645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 2 [ 1126.110489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_interrupt -r 2 [ 1126.149935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_interrupt -r 2 [ 1126.188560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_interrupt -r 2 [ 1126.211176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_interrupt -r 2 [ 1126.248582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 1126.267124] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 1126.270898] LNet: 1 local NIs in recovery (showing 1): 192.168.203.25@tcp [ 1126.567681] LNet: 78278:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.648363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.684215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.133570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_dropped -r 2 [ 1127.162829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 2 [ 1127.193891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_dropped -r 2 [ 1127.231436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 2 [ 1127.274430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_dropped -r 2 [ 1127.313321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_dropped -r 2 [ 1127.344524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_dropped -r 2 [ 1127.375894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_dropped -r 2 [ 1127.410582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 1127.436602] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 1127.728119] LNet: 78386:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.814323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.855448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.322982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_aborted -r 2 [ 1128.348296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 2 [ 1128.370163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_aborted -r 2 [ 1128.396420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 2 [ 1128.425609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_aborted -r 2 [ 1128.468236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_aborted -r 2 [ 1128.498564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_aborted -r 2 [ 1128.530859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_aborted -r 2 [ 1128.563549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 1128.582900] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 1128.852334] LNet: 78494:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.909043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.937752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.424650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_no_route -r 2 [ 1129.461700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 2 [ 1129.483643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_no_route -r 2 [ 1129.520602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 2 [ 1129.565274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_no_route -r 2 [ 1129.617840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_no_route -r 2 [ 1129.676966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_no_route -r 2 [ 1129.720154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_no_route -r 2 [ 1129.748927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 1130.118423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.147868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.634061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e local_timeout -r 2 [ 1130.671707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 2 [ 1130.702245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e local_timeout -r 2 [ 1130.728332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 2 [ 1130.763934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e local_timeout -r 2 [ 1130.795236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e local_timeout -r 2 [ 1130.840269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e local_timeout -r 2 [ 1130.873423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e local_timeout -r 2 [ 1130.917237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 1130.937113] LNet: There was an unexpected network error while writing to 192.168.203.125: rc = -22 [ 1130.940347] LNet: Skipped 1 previous similar message [ 1131.173863] LNet: 78711:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.179160] LNet: 78711:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1131.230472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.266823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.670977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp -m GET -e remote_dropped -r 2 [ 1131.703616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 2 [ 1131.733696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp -m GET -e remote_dropped -r 2 [ 1131.758176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.25@tcp -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 2 [ 1131.788121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp -m GET -e remote_dropped -r 2 [ 1131.815329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.125@tcp1 -m GET -e remote_dropped -r 2 [ 1131.861950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp -m GET -e remote_dropped -r 2 [ 1131.896796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.25@tcp1 -d 192.168.203.25@tcp1 -m GET -e remote_dropped -r 2 [ 1131.942567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.25@tcp 192.168.203.125@tcp [ 1131.961177] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 1132.319879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.358527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.307559] LNet: 78887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.315093] LNetError: 78887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.322763] LNet: Removed LNI 192.168.203.25@tcp [ 1136.335245] LNet: Removed LNI 192.168.203.25@tcp1 [ 1136.719411] Key type .llcrypt unregistered [ 1136.721254] Key type ._llcrypt unregistered [ 1148.317481] Key type ._llcrypt registered [ 1148.319264] Key type .llcrypt registered [ 1149.519988] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:36:41 (1762490201) [ 1149.771938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.780588] alg: No test for adler32 (adler32-zlib) [ 1150.732847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.802509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1150.835172] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1151.634106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1151.689748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1151.740687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1151.801459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1151.838873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1151.887676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1157.076860] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:36:49 (1762490209) [ 1157.211750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1157.249039] LNet: 80440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.258249] LNetError: 80440:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.272231] LNet: Removed LNI 192.168.203.25@tcp [ 1157.297865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1157.357901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1157.389703] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1162.990947] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:36:55 (1762490215) [ 1163.077119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.113287] LNet: 81133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.122290] LNetError: 81133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.136239] LNet: Removed LNI 192.168.203.25@tcp [ 1163.160437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.243631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1163.291508] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1169.832938] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:37:02 (1762490222) [ 1171.014312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1172.484083] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:37:04 (1762490224) [ 1172.802545] LNet: 82172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.808173] LNetError: 82172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.820167] LNet: Removed LNI 192.168.203.25@tcp [ 1173.260578] Key type .llcrypt unregistered [ 1173.263107] Key type ._llcrypt unregistered [ 1177.624033] Key type ._llcrypt registered [ 1177.631418] Key type .llcrypt registered [ 1177.789087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.806928] alg: No test for adler32 (adler32-zlib) [ 1178.667733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1178.692957] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1184.952509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1189.257446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.125@tcp [ 1189.491806] LNet: 82583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.497540] LNetError: 82583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.511718] LNet: Removed LNI 192.168.203.25@tcp [ 1189.890518] Key type .llcrypt unregistered [ 1189.895298] Key type ._llcrypt unregistered [ 1197.882150] Key type ._llcrypt registered [ 1197.883630] Key type .llcrypt registered [ 1199.180975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1200.379310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1201.952884] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:37:34 (1762490254) [ 1202.343971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1202.355741] alg: No test for adler32 (adler32-zlib) [ 1203.225071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1203.278082] LNet: 83668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.743645] Key type .llcrypt unregistered [ 1203.748527] Key type ._llcrypt unregistered [ 1207.377462] Key type ._llcrypt registered [ 1207.379520] Key type .llcrypt registered [ 1207.502940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.510537] alg: No test for adler32 (adler32-zlib) [ 1208.386602] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1208.388749] LNet: 83920:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1208.427881] LNetError: 83932:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 1208.453626] LNetError: 83932:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.29@tcp has gone from up to down [ 1208.739124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.758986] LNet: 83978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.766322] LNetError: 83978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.783454] LNet: Removed LNI 192.168.203.25@tcp [ 1208.804964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.848751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1208.867878] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1208.872546] LNetError: 83985:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.27@tcp is a gateway. Can not delete it [ 1208.879348] LNetError: 83985:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.203.27@tcp is a router. Can not be deleted [ 1208.895483] LNet: 83985:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.657832] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:37:46 (1762490266) [ 1214.953904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg325-server [ 1216.333608] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:37:48 (1762490268) [ 1217.806219] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg325-server [ 1218.999279] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:37:51 (1762490271) [ 1220.358236] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1222.168360] Key type lgssc registered [ 1223.395770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1238.458183] Key type lgssc unregistered [ 1238.745935] LNet: 86401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.751417] LNetError: 86401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.776885] LNet: Removed LNI 192.168.203.25@tcp [ 1239.178261] Key type .llcrypt unregistered [ 1239.179855] Key type ._llcrypt unregistered [ 1249.915361] Key type ._llcrypt registered [ 1249.917112] Key type .llcrypt registered [ 1250.911489] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:38:23 (1762490303) [ 1251.278813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.292275] alg: No test for adler32 (adler32-zlib) [ 1252.211751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.260211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1252.290896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1252.331147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1252.360973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1257.830627] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:38:30 (1762490310) [ 1257.934061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.975701] LNet: 87796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.002948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.078969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1258.105625] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1258.823480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1258.846885] LNet: 87944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.850713] LNetError: 87944:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.865622] LNet: Removed LNI 192.168.203.25@tcp [ 1258.887949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.931705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1258.952846] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1259.576629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1259.608172] LNetError: 87972:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.639250] LNet: Removed LNI 192.168.203.25@tcp [ 1259.665662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1259.774503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1259.847092] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1266.276305] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:38:38 (1762490318) [ 1266.373397] LNet: 88554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.379632] LNet: 88554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1266.387866] LNetError: 88554:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.408542] LNet: Removed LNI 192.168.203.25@tcp [ 1267.019913] Key type .llcrypt unregistered [ 1267.034328] Key type ._llcrypt unregistered [ 1271.908241] Key type ._llcrypt registered [ 1271.909627] Key type .llcrypt registered [ 1272.077674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.096352] alg: No test for adler32 (adler32-zlib) [ 1272.952535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.064175] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1273.068125] LNet: *** cfs_fail_loc=e003, val=100*** [ 1273.703904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1273.732873] LNet: *** cfs_fail_loc=e003, val=100*** [ 1273.738822] LNet: Skipped 2 previous similar messages [ 1273.752868] Ping reply received with mapping: [ 1273.754145] found metadata reply NI[192.168.203.25@tcp] [ 1273.760642] 19 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1273.770171] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1273.780197] 5a 5a 5a 5a 5a 5a 5a 5a [ 1273.895040] LNet: 88951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.904135] LNetError: 88951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.920801] LNet: Removed LNI 192.168.203.25@tcp [ 1274.550174] Key type .llcrypt unregistered [ 1274.552532] Key type ._llcrypt unregistered [ 1282.259834] Key type ._llcrypt registered [ 1282.261485] Key type .llcrypt registered [ 1283.130533] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:38:55 (1762490335) [ 1283.461530] Key type .llcrypt unregistered [ 1283.463360] Key type ._llcrypt unregistered [ 1287.202590] Key type ._llcrypt registered [ 1287.211866] Key type .llcrypt registered [ 1287.414633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1287.425212] alg: No test for adler32 (adler32-zlib) [ 1288.475724] LNetError: 89986:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1288.830297] Key type .llcrypt unregistered [ 1288.840357] Key type ._llcrypt unregistered [ 1293.683774] Key type ._llcrypt registered [ 1293.685775] Key type .llcrypt registered [ 1294.842088] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:39:07 (1762490347) [ 1295.268433] Key type .llcrypt unregistered [ 1295.271357] Key type ._llcrypt unregistered [ 1300.152662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1300.156811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1302.690688] Key type ._llcrypt registered [ 1302.698395] Key type .llcrypt registered [ 1302.910463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1302.936331] alg: No test for adler32 (adler32-zlib) [ 1303.830831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.887219] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1311.925596] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:39:24 (1762490364) [ 1312.031792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1312.083337] LNet: 91755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.096387] LNetError: 91755:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1312.135122] LNet: Removed LNI 10.1.2.3@tcp [ 1312.175843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.950698] Key type .llcrypt unregistered [ 1312.954447] Key type ._llcrypt unregistered [ 1317.772816] Key type ._llcrypt registered [ 1317.774278] Key type .llcrypt registered [ 1317.942073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1317.961068] alg: No test for adler32 (adler32-zlib) [ 1318.838179] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1318.887431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1319.011708] LNet: 92042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.019762] LNetError: 92042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.035892] LNet: Removed LNI 192.168.203.25@tcp [ 1319.474571] Key type .llcrypt unregistered [ 1319.477490] Key type ._llcrypt unregistered [ 1327.495507] Key type ._llcrypt registered [ 1327.497826] Key type .llcrypt registered [ 1328.447544] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:39:40 (1762490380) [ 1328.877747] Key type .llcrypt unregistered [ 1328.884250] Key type ._llcrypt unregistered [ 1332.879772] Key type ._llcrypt registered [ 1332.882922] Key type .llcrypt registered [ 1333.113437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1333.129160] alg: No test for adler32 (adler32-zlib) [ 1339.631268] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:39:51 (1762490391) [ 1339.726826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.783672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.855261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.932569] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1340.661378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.701724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1340.728997] LNet: Removed LNI 192.168.203.25@tcp [ 1340.925868] LNet: 93774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.259471] Key type .llcrypt unregistered [ 1341.261379] Key type ._llcrypt unregistered [ 1348.999332] Key type ._llcrypt registered [ 1349.000763] Key type .llcrypt registered [ 1350.015946] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:40:02 (1762490402) [ 1350.444299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1350.456361] alg: No test for adler32 (adler32-zlib) [ 1351.347458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.395637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1351.428556] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1351.989450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1352.024895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.25@tcp [ 1352.036914] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1352.040158] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1352.043493] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1352, 0, 0 [ 1352.047785] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1352.049450] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1352.052068] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1352.058364] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1352, 0, 0 [ 1352.061756] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1352.064173] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1352.067059] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1352.069082] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1352, 0, 1402 [ 1352.071508] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1352.073375] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1352.075685] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1352.079792] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1352, 0, 1402 [ 1352.083443] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1356.731486] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:40:09 (1762490409) [ 1357.030856] LNet: 95308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.038939] LNetError: 95308:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.057906] LNet: Removed LNI 192.168.203.25@tcp [ 1357.551493] Key type .llcrypt unregistered [ 1357.553281] Key type ._llcrypt unregistered [ 1360.886617] Key type ._llcrypt registered [ 1360.891748] Key type .llcrypt registered [ 1361.030621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1361.043127] alg: No test for adler32 (adler32-zlib) [ 1361.894289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1361.913107] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1366.820686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1373.111432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.173208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1373.204198] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1374.020170] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1374.113151] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1374.116638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1374.120388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1376.418767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1376.437871] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1383.945525] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:40:36 (1762490436) [ 1384.111728] LNet: 96802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.121567] LNetError: 96802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.126890] LNet: Removed LNI 192.168.203.25@tcp [ 1385.184134] LNet: Removed LNI 192.168.203.25@tcp1 [ 1385.186130] LNetError: 96802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1385.208334] LNetError: 96802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1385.226382] LNet: Removed LNI 10.1.2.3@tcp99 [ 1385.838957] Key type .llcrypt unregistered [ 1385.840745] Key type ._llcrypt unregistered [ 1390.321460] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1390.364568] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1392.758051] Key type ._llcrypt registered [ 1392.759677] Key type .llcrypt registered [ 1392.911452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1392.924647] alg: No test for adler32 (adler32-zlib) [ 1393.820784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.863934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.887173] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1394.491753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1394.514891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1396.175108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1396.224486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1396.258213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1396.269701] LNetError: 97535:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1396.295773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1396.328675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1396.621108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1396.673501] LNet: 97555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.691902] LNetError: 97555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.699199] LNet: Removed LNI 192.168.203.25@tcp [ 1396.708902] LNet: Removed LNI 10.1.2.3@tcp [ 1397.218145] Key type .llcrypt unregistered [ 1397.220284] Key type ._llcrypt unregistered [ 1404.384471] Key type ._llcrypt registered [ 1404.386150] Key type .llcrypt registered [ 1405.355379] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:40:57 (1762490457) [ 1405.574896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.584018] alg: No test for adler32 (adler32-zlib) [ 1406.492404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.543399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.566151] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1407.192174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.25@tcp [ 1407.242404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg325-client@tcp [ 1412.540337] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:41:04 (1762490464) [ 1412.623071] LNet: 99072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.629794] LNetError: 99072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.643155] LNet: Removed LNI 192.168.203.25@tcp [ 1413.034534] Key type .llcrypt unregistered [ 1413.036182] Key type ._llcrypt unregistered [ 1416.954559] Key type ._llcrypt registered [ 1416.956216] Key type .llcrypt registered [ 1417.093264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.107947] alg: No test for adler32 (adler32-zlib) [ 1418.010264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1418.038414] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1418.066898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1422.911138] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:41:15 (1762490475) [ 1424.016717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1425.476737] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:41:17 (1762490477) [ 1426.796189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1428.381280] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:41:20 (1762490480) [ 1428.485614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.506613] LNet: 100488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.515476] LNetError: 100488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.537878] LNet: Removed LNI 192.168.203.25@tcp [ 1428.552958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.589519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.618908] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1429.241547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1429.279700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1434.668525] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:41:27 (1762490487) [ 1434.769370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.784277] LNet: 101198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.788220] LNetError: 101198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.800126] LNet: Removed LNI 192.168.203.25@tcp [ 1434.853653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.892932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1434.923888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1435.399337] Key type .llcrypt unregistered [ 1435.400967] Key type ._llcrypt unregistered [ 1442.550683] Key type ._llcrypt registered [ 1442.552013] Key type .llcrypt registered [ 1443.383222] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:41:35 (1762490495) [ 1443.691233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.702539] alg: No test for adler32 (adler32-zlib) [ 1444.554470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.602907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.626201] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1445.055650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1445.091495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1449.683397] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:41:42 (1762490502) [ 1449.843860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.862861] LNet: 102749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.868815] LNetError: 102749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.880661] LNet: Removed LNI 192.168.203.25@tcp [ 1449.899613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.948209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1449.975559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1453.918543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1454.996745] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:41:47 (1762490507) [ 1455.138792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.152903] LNet: 103468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.171413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.371424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.396119] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1455.910640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1455.930952] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1456.598558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.25@tcp [ 1456.635809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1456.685724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.25@tcp -m PUT -e local_timeout -r 1 [ 1456.748198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1456.795561] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1456.797919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1456.967914] LNetError: 103500:0:(lib-move.c:4947:lnet_parse()) 192.168.203.25@tcp, src 192.168.203.25@tcp: Dropping PUT (error -108 looking up sender) [ 1457.829108] LNet: 103775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.836369] LNetError: 103775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.840784] LNet: Removed LNI 192.168.203.25@tcp [ 1457.850417] LNet: Removed LNI 10.1.2.3@tcp [ 1458.269070] Key type .llcrypt unregistered [ 1458.270934] Key type ._llcrypt unregistered [ 1467.354398] Key type ._llcrypt registered [ 1467.355925] Key type .llcrypt registered [ 1468.447589] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:42:00 (1762490520) [ 1468.879953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1468.891105] alg: No test for adler32 (adler32-zlib) [ 1469.850872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.107560] LNet: 104623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.730182] Key type .llcrypt unregistered [ 1470.736731] Key type ._llcrypt unregistered [ 1475.122253] Key type ._llcrypt registered [ 1475.124590] Key type .llcrypt registered [ 1475.268866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1475.296506] alg: No test for adler32 (adler32-zlib) [ 1476.186001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1476.234881] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1482.238938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1482.438125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1483.011533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.125@tcp [ 1483.396784] LNet: 105061:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1483.425824] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 1483.477370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1483.515230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1484.126690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1484.439225] LNet: 105166:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1484.503914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1484.538929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1485.051908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.125@tcp [ 1485.514174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.541442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.174919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1486.507455] LNet: 105377:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.513732] LNet: 105377:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1486.593411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.614253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.869445] LNet: 105418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.878430] LNetError: 105418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.891758] LNet: Removed LNI 192.168.203.25@tcp [ 1487.235510] Key type .llcrypt unregistered [ 1487.238144] Key type ._llcrypt unregistered [ 1501.217932] Key type ._llcrypt registered [ 1501.219452] Key type .llcrypt registered [ 1502.179672] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:42:34 (1762490554) [ 1502.483245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.497913] alg: No test for adler32 (adler32-zlib) [ 1503.375612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.682753] LNet: 106317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.240443] Key type .llcrypt unregistered [ 1504.245120] Key type ._llcrypt unregistered [ 1508.333869] Key type ._llcrypt registered [ 1508.344551] Key type .llcrypt registered [ 1508.532779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1508.549992] alg: No test for adler32 (adler32-zlib) [ 1509.422320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1509.460585] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1514.905789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.125@tcp [ 1520.534181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.570177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.597920] LNet: Added LNI 192.168.203.25@tcp1 [8/256/0/180] [ 1521.201984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1521.591239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.125@tcp [ 1521.760089] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.125@tcp [ 1521.877220] LNet: 106953:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1521.929503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1521.955930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1522.374330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1522.614659] LNet: 107058:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1522.671814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1522.703115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1523.163924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.125@tcp [ 1523.422880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1523.453679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1523.904995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1524.175935] LNet: 107268:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1524.179429] LNet: 107268:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1524.240936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1524.282453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.710427] LNet: 107343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.719498] LNetError: 107343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.778378] LNet: Removed LNI 192.168.203.25@tcp [ 1527.780174] LNetError: 107343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.802337] LNet: Removed LNI 192.168.203.25@tcp1 [ 1528.247591] Key type .llcrypt unregistered [ 1528.250324] Key type ._llcrypt unregistered [ 1539.481963] Key type ._llcrypt registered [ 1539.483587] Key type .llcrypt registered [ 1540.487764] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1353 sec ========== 23:43:12 (1762490592) [ 1541.097244] Key type .llcrypt unregistered [ 1541.098929] Key type ._llcrypt unregistered [ 1549.680663] Key type ._llcrypt registered [ 1549.682214] Key type .llcrypt registered [ 1549.905906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1549.918450] alg: No test for adler32 (adler32-zlib) [ 1550.874880] Lustre: Lustre: Build Version: 2.16.58_106_gac70ede [ 1551.085381] LNet: Added LNI 192.168.203.25@tcp [8/256/0/180] [ 1552.745318] Key type lgssc registered [ 1553.664935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1600.595348] Lustre: Mounted lustre-client [ 1604.972932] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1615.710706] LustreError: 110567:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e2d8a374800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1615.720867] LustreError: 110567:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1615.740821] Lustre: Unmounted lustre-client [ 1634.713356] Key type lgssc unregistered [ 1634.921703] LNet: 111048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.925880] LNetError: 111048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.956537] LNet: Removed LNI 192.168.203.25@tcp [ 1635.504172] Key type .llcrypt unregistered [ 1635.505827] Key type ._llcrypt unregistered