[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485398623 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003281] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008412] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012040] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.022443] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026598] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027225] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030067] *** VALIDATE tmpfs *** [ 0.031388] *** VALIDATE proc *** [ 0.032213] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034307] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.040000] debug: unmapping init [mem 0xffffffff89c59000-0xffffffff89c60fff] [ 0.041927] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043034] ... version: 2 [ 0.044019] ... bit width: 48 [ 0.045020] ... generic registers: 4 [ 0.046017] ... value mask: 0000ffffffffffff [ 0.047019] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049017] ... event mask: 000000070000000f [ 0.051207] rcu: Hierarchical SRCU implementation. [ 0.053592] smp: Bringing up secondary CPUs ... [ 0.054623] x86: Booting SMP configuration: [ 0.055035] .... node #0, CPUs: #1 #2 #3 [ 0.060050] smp: Brought up 1 node, 4 CPUs [ 0.062022] smpboot: Max logical packages: 1 [ 0.063033] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231044] node 0 deferred pages initialised in 164ms [ 0.233165] devtmpfs: initialized [ 0.234234] x86/mm: Memory block size: 128MB [ 0.238899] gcov: version magic: 0x41383552 [ 0.241680] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246245] pinctrl core: initialized pinctrl subsystem [ 0.248180] [ 0.248761] ************************************************************* [ 0.251014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253011] ** ** [ 0.256012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258009] ** ** [ 0.260014] ** This means that this kernel is built to expose internal ** [ 0.262013] ** IOMMU data structures, which may compromise security on ** [ 0.265014] ** your system. ** [ 0.266008] ** ** [ 0.268011] ** If you see this message and you are not debugging the ** [ 0.270013] ** kernel, report this immediately to your vendor! ** [ 0.272012] ** ** [ 0.275013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277011] ************************************************************* [ 0.278591] NET: Registered protocol family 16 [ 0.280422] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.285051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288217] cpuidle: using governor menu [ 0.290000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.293288] PCI: Using configuration type 1 for base access [ 0.297125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.310124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.312042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315238] cryptd: max_cpu_qlen set to 1000 [ 0.318275] ACPI: Added _OSI(Module Device) [ 0.321020] ACPI: Added _OSI(Processor Device) [ 0.323012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.325013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.331000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.338486] ACPI: Interpreter enabled [ 0.340053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.341011] ACPI: Using IOAPIC for interrupt routing [ 0.343082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.346731] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.359608] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.362035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.364015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.367077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.372570] acpiphp: Slot [2] registered [ 0.374161] acpiphp: Slot [3] registered [ 0.376085] acpiphp: Slot [4] registered [ 0.377115] acpiphp: Slot [5] registered [ 0.379121] acpiphp: Slot [6] registered [ 0.380107] acpiphp: Slot [7] registered [ 0.381100] acpiphp: Slot [8] registered [ 0.383056] acpiphp: Slot [9] registered [ 0.385133] acpiphp: Slot [10] registered [ 0.387110] acpiphp: Slot [11] registered [ 0.389103] acpiphp: Slot [12] registered [ 0.392465] acpiphp: Slot [13] registered [ 0.393085] acpiphp: Slot [14] registered [ 0.395105] acpiphp: Slot [15] registered [ 0.397121] acpiphp: Slot [16] registered [ 0.399123] acpiphp: Slot [17] registered [ 0.401093] acpiphp: Slot [18] registered [ 0.403165] acpiphp: Slot [19] registered [ 0.405104] acpiphp: Slot [20] registered [ 0.406071] acpiphp: Slot [21] registered [ 0.407079] acpiphp: Slot [22] registered [ 0.409401] acpiphp: Slot [23] registered [ 0.411096] acpiphp: Slot [24] registered [ 0.412091] acpiphp: Slot [25] registered [ 0.414104] acpiphp: Slot [26] registered [ 0.415102] acpiphp: Slot [27] registered [ 0.416087] acpiphp: Slot [28] registered [ 0.418095] acpiphp: Slot [29] registered [ 0.419084] acpiphp: Slot [30] registered [ 0.421091] acpiphp: Slot [31] registered [ 0.422076] PCI host bridge to bus 0000:00 [ 0.424018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.426021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.428021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.431021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433021] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.435022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.437173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.441279] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.446584] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.453000] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.457049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.460018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.462016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.464018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.466528] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.470069] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.472048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.474627] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.477025] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.484018] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.488970] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.494444] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.502018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.507015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.516996] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.524251] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.531015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.546028] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.555308] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.557316] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.559302] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.562316] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.564180] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.569170] iommu: Default domain type: Passthrough [ 0.570395] SCSI subsystem initialized [ 0.573152] ACPI: bus type USB registered [ 0.574072] usbcore: registered new interface driver usbfs [ 0.577127] usbcore: registered new interface driver hub [ 0.579106] usbcore: registered new device driver usb [ 0.580137] pps_core: LinuxPPS API ver. 1 registered [ 0.582014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.585058] PTP clock support registered [ 0.587101] EDAC MC: Ver: 3.0.0 [ 0.589149] PCI: Using ACPI for IRQ routing [ 0.591078] NetLabel: Initializing [ 0.592015] NetLabel: domain hash size = 128 [ 0.594018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.596263] NetLabel: unlabeled traffic allowed by default [ 0.599440] vgaarb: loaded [ 0.602437] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.604017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.611331] clocksource: Switched to clocksource kvm-clock [ 0.722215] VFS: Disk quotas dquot_6.6.0 [ 0.724253] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.727389] *** VALIDATE ramfs *** [ 0.730307] *** VALIDATE hugetlbfs *** [ 0.731862] pnp: PnP ACPI init [ 0.734265] pnp: PnP ACPI: found 6 devices [ 0.757707] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.761750] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.763824] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.766594] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.769016] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.771035] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.773383] NET: Registered protocol family 2 [ 0.775867] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.781030] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.784823] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.789581] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.793028] TCP: Hash tables configured (established 65536 bind 65536) [ 0.795407] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.798648] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.801153] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803696] NET: Registered protocol family 1 [ 0.806529] RPC: Registered named UNIX socket transport module. [ 0.810206] RPC: Registered udp transport module. [ 0.811894] RPC: Registered tcp transport module. [ 0.813649] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.816267] NET: Registered protocol family 44 [ 0.817817] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.820600] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.823502] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.826017] PCI: CLS 0 bytes, default 64 [ 0.828271] Unpacking initramfs... [ 2.255914] debug: unmapping init [mem 0xffff93e77cc64000-0xffff93e77ffcffff] [ 2.259963] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.261858] software IO TLB: mapped [mem 0x00000000a0600000-0x00000000a4600000] (64MB) [ 2.265076] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.806416] Initialise system trusted keyrings [ 2.808226] Key type blacklist registered [ 2.810367] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.821464] zbud: loaded [ 2.824606] *** VALIDATE nfs *** [ 2.825892] *** VALIDATE nfs4 *** [ 2.827354] pstore: using deflate compression [ 2.831244] Platform Keyring initialized [ 2.974893] NET: Registered protocol family 38 [ 2.976704] Key type asymmetric registered [ 2.978747] Asymmetric key parser 'x509' registered [ 2.980922] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.984180] io scheduler mq-deadline registered [ 2.985971] io scheduler kyber registered [ 2.987645] io scheduler bfq registered [ 2.989435] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.992732] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.001438] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.003725] ACPI: Power Button [PWRF] [ 3.101879] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.194706] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.294973] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.334802] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.365692] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.370595] Non-volatile memory driver v1.3 [ 3.372220] Linux agpgart interface v0.103 [ 3.408365] virtio_blk virtio1: [vda] 133776 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.412464] vda: detected capacity change from 0 to 68493312 [ 3.429086] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.432113] vdb: detected capacity change from 0 to 1073741824 [ 3.438311] libphy: Fixed MDIO Bus: probed [ 3.449264] usbcore: registered new interface driver usbserial_generic [ 3.451357] usbserial: USB Serial support registered for generic [ 3.453255] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.456512] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.458948] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.460767] mousedev: PS/2 mouse device common for all mice [ 3.463910] rtc_cmos 00:05: RTC can wake from S4 [ 3.467254] rtc_cmos 00:05: registered as rtc0 [ 3.470943] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.474322] intel_pstate: CPU model not supported [ 3.474521] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.479490] hid: raw HID events driver (C) Jiri Kosina [ 3.481955] usbcore: registered new interface driver usbhid [ 3.483848] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.484760] usbhid: USB HID core driver [ 3.489220] drop_monitor: Initializing network drop monitor service [ 3.489266] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.491185] Initializing XFRM netlink socket [ 3.491651] NET: Registered protocol family 10 [ 3.499084] Segment Routing with IPv6 [ 3.500817] NET: Registered protocol family 17 [ 3.503044] mpls_gso: MPLS GSO support [ 3.509417] RAS: Correctable Errors collector initialized. [ 3.511633] AVX version of gcm_enc/dec engaged. [ 3.515670] AES CTR mode by8 optimization enabled [ 3.620404] sched_clock: Marking stable (3620366911, 0)->(4495719465, -875352554) [ 3.626377] registered taskstats version 1 [ 3.629048] Loading compiled-in X.509 certificates [ 3.631586] zswap: loaded using pool lzo/zbud [ 3.671256] Key type big_key registered [ 3.688499] Key type encrypted registered [ 3.691527] ima: No TPM chip found, activating TPM-bypass! [ 3.694241] ima: Allocated hash algorithm: sha1 [ 3.696320] ima: No architecture policies found [ 3.698195] evm: Initialising EVM extended attributes: [ 3.699860] evm: security.selinux [ 3.701328] evm: security.ima [ 3.702504] evm: security.capability [ 3.703812] evm: HMAC attrs: 0x1 [ 3.706546] rtc_cmos 00:05: setting system clock to 2025-10-23 16:42:04 UTC (1761237724) [ 3.712610] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.716184] debug: unmapping init [mem 0xffffffff89982000-0xffffffff89c58fff] [ 3.725117] Write protecting the kernel read-only data: 28672k [ 3.729880] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.736926] debug: unmapping init [mem 0xffffffff88914000-0xffffffff889fffff] [ 3.775107] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.783391] systemd[1]: Detected virtualization kvm. [ 3.785153] systemd[1]: Detected architecture x86-64. [ 3.787115] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.814450] systemd[1]: No hostname configured. [ 3.816243] systemd[1]: Set hostname to . [ 3.818251] random: systemd: uninitialized urandom read (16 bytes read) [ 3.820825] systemd[1]: Initializing machine ID from random generator. [ 3.870959] random: ln: uninitialized urandom read (6 bytes read) [ 3.989512] random: systemd: uninitialized urandom read (16 bytes read) [ 3.993335] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.999465] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.005930] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.762145] device-mapper: uevent: version 1.0.3 [ 4.764836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.642955] random: fast init done [ 5.653451] virtio_net virtio0 ens2: renamed from eth0 [ 5.711974] scsi host0: ata_piix [ 5.738871] scsi host1: ata_piix [ 5.740613] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.743124] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.568325] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.209656] random: crng init done [ 10.210750] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.925730] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.235667] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.510271] SELinux: Disabled at runtime. [ 12.566341] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.575328] systemd[1]: Detected virtualization kvm. [ 12.577225] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.134481] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.138586] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.146769] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.152288] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.156870] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.163496] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.171474] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ 13.220830] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.700597] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.046186] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.051968] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.158092] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.169643] EDAC sbridge: Ver: 1.1.2 [ 15.336858] Key type dns_resolver registered [ 15.642483] NFS: Registering the id_resolver key type [ 15.643911] Key type id_resolver registered [ 15.645427] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg358-client login: [ 61.284539] libcfs: loading out-of-tree module taints kernel. [ 61.337570] Key type ._llcrypt registered [ 61.338828] Key type .llcrypt registered [ 61.838672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 61.855852] alg: No test for adler32 (adler32-zlib) [ 63.120543] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 64.006867] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 65.839299] Key type lgssc registered [ 67.482832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.348995] hrtimer: interrupt took 8706272 ns [ 177.921264] Lustre: Mounted lustre-client [ 183.144353] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.990681] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing check_logdir /tmp/testlogs/ [ 203.562563] Lustre: DEBUG MARKER: oleg358-client.virtnet: executing yml_node [ 203.743180] Lustre: lustre-OST0000-osc-ffff93e7d8e9f000: disconnect after 23s idle [ 207.499673] Lustre: DEBUG MARKER: Client: 2.16.59.36 [ 209.932618] Lustre: DEBUG MARKER: MDS: 2.16.59.36 [ 212.612597] Lustre: DEBUG MARKER: OSS: 2.16.59.36 [ 214.004989] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Oct 23 12:45:33 EDT 2025 [ 227.044389] LustreError: 7903:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff93e7d8e9f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 227.051642] LustreError: 7903:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 227.083187] Lustre: Unmounted lustre-client [ 260.163888] Key type lgssc unregistered [ 260.437911] LNet: 8351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.450289] LNetError: 8351:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 260.469113] LNet: Removed LNI 192.168.203.58@tcp [ 260.996187] Key type .llcrypt unregistered [ 260.999947] Key type ._llcrypt unregistered [ 280.406347] Key type ._llcrypt registered [ 280.408181] Key type .llcrypt registered [ 280.570650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 280.581791] alg: No test for adler32 (adler32-zlib) [ 281.451479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 281.504576] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 281.517350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 284.663370] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 284.800302] LNet: 9221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 284.804098] LNetError: 9221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 284.817843] LNet: Removed LNI 192.168.203.58@tcp [ 285.387827] Key type .llcrypt unregistered [ 285.389328] Key type ._llcrypt unregistered [ 290.563968] Key type ._llcrypt registered [ 290.570603] Key type .llcrypt registered [ 291.763869] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:46:51 (1761238011) [ 292.172534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 292.187479] alg: No test for adler32 (adler32-zlib) [ 293.063482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.125581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.141933] LNet: 9630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.162763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.209329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 293.230426] LNet: 9637:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 298.798972] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:46:58 (1761238018) [ 298.864983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.902455] LNet: 10207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.927710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.010515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 304.319867] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:47:03 (1761238023) [ 304.474178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.516635] LNet: 10798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.543300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.610720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 310.218435] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:47:09 (1761238029) [ 310.601710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.646977] LNet: 11383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.694901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.808578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 316.933355] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:47:16 (1761238036) [ 317.126905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.147355] LNet: 11967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.185140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.268440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 317.537096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 323.101946] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:47:22 (1761238042) [ 323.239807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.267349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.315614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 323.688756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 323.979822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 329.536748] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:47:28 (1761238048) [ 329.643937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.690901] LNet: 13148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.694128] LNet: 13148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 329.745868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.935047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 337.648043] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:47:37 (1761238057) [ 337.840577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.883766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.941300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 337.981040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 338.363508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 338.447128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 338.956688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 338.990695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 339.362347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 339.385048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 339.703142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 339.740533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 340.200401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 340.229798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 340.571905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 340.619198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 341.016298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 341.046574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 341.430836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 341.455333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 341.863031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 341.897560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 342.457250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 342.488311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 342.817887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 342.855509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 343.262929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 343.311451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 349.622942] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:47:49 (1761238069) [ 349.750681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.778326] LNet: 14417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.781723] LNet: 14417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 349.809664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.875845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 349.924676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 357.340301] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:47:56 (1761238076) [ 357.423232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.497181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.596416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 357.636965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 363.663494] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:48:03 (1761238083) [ 363.884628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.911084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.966641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 363.987309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 370.014330] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:48:09 (1761238089) [ 370.176821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.222515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.305107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 370.335333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 376.405269] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:48:16 (1761238096) [ 376.506801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.571915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.653340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 376.691398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 382.488902] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:48:22 (1761238102) [ 382.795723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.831556] LNet: 17355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.839609] LNet: 17355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 382.856265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.991510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 383.024386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 389.191096] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:48:28 (1761238108) [ 389.557409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.597078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.672162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 390.105219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 397.067496] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:48:36 (1761238116) [ 397.436478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.499244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.670931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 397.896492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 404.140241] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:48:43 (1761238123) [ 404.296022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.360339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.492242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.750608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 411.495838] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:48:50 (1761238130) [ 411.799661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.841732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.361206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.582917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.107403] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:48:58 (1761238138) [ 419.361402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.400554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.468943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 419.534930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 425.846590] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:49:05 (1761238145) [ 426.007230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.039215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.084752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.128928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.107801] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:49:11 (1761238151) [ 432.232685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.282552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.348488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 438.375455] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:49:17 (1761238157) [ 438.603928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.668587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.747419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 444.993957] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:49:24 (1761238164) [ 445.096931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.141724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.251985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 445.657797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 450.940706] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:49:30 (1761238170) [ 451.175251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.199222] LNet: 23381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.209262] LNet: 23381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 451.236671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.299444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 451.320879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 457.086549] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:49:36 (1761238176) [ 457.205378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.270756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.357208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 457.389133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 463.735594] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:49:43 (1761238183) [ 464.099327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.141978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.229080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 464.264660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 470.619676] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:49:50 (1761238190) [ 470.717820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.780054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.873390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 470.924631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 477.459680] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:49:57 (1761238197) [ 477.588113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.645439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.786474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 477.837708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 477.893695] LNetError: 25738:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 477.918973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 477.960522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 483.681225] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:50:03 (1761238203) [ 483.925531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.952157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.025817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Mxc8umTu8U [ 484.104846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Mxc8umTu8U [ 484.162770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Mxc8umTu8U [ 489.083886] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:50:08 (1761238208) [ 489.372285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.414572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.467456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 489.499181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 489.557290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 494.407363] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:50:14 (1761238214) [ 494.514726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.571371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.684406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 496.646760] LNet: Message forwarding will be enabled in 2s [ 499.743116] LNet: Message forwarding enabled [ 500.004782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 500.042378] LNet: Message forwarding will stop in 2s to 6s [ 504.547202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 505.063585] LNet: Message forwarding disabled [ 506.535452] LNet: Message forwarding will be enabled in 2s [ 510.177044] LNet: Message forwarding enabled [ 511.324770] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:50:31 (1761238231) [ 511.444733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.350708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.396800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 516.941472] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:50:36 (1761238236) [ 517.114455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.160970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.207627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 517.236979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 517.265949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 517.303452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 517.335536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 517.358573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 517.378428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 517.407960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 517.433171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 517.463724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 517.488849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 517.515617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 517.548933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 517.575780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 517.605827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 517.654880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 517.685182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 517.718707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 517.768608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 517.782831] LNetError: 28807:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 522.421268] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:50:42 (1761238242) [ 522.583779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.603540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.650404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 527.886147] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:50:47 (1761238247) [ 529.303846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg358-server [ 531.046980] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:50:50 (1761238250) [ 531.288330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.325783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.358628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 531.379371] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 533.005512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[59-63]@tcp [ 533.041409] LNet: 30519:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 533.105357] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 533.116834] LNetError: 30312:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.60@tcp has gone from up to down [ 534.134562] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 534.154119] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 535.158644] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 535.168732] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 536.160578] LNetError: 30317:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 536.177403] LNetError: 30317:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 537.254142] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 537.277901] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 538.763451] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:50:58 (1761238258) [ 540.097879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg358-server [ 541.357843] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 541.361936] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:51:01 (1761238261) [ 541.366657] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 541.506540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.367833] LNetError: 30311:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.203.59@tcp [ 542.560968] LNetError: 31452:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.581401] LNet: Removed LNI 192.168.203.58@tcp [ 542.601361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.640909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.662418] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 543.206561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[59-64/2]@tcp [ 543.228730] LNet: 31568:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 543.233368] LNet: 31568:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 543.256619] LNetError: 31458:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.59@tcp has gone from up to down [ 543.258780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[59-64/2]@tcp [ 543.265374] LNetError: 31458:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 548.908522] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:51:08 (1761238268) [ 548.996404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.021601] LNetError: 32152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.032779] LNet: Removed LNI 192.168.203.58@tcp [ 549.053451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.117652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 549.169141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.209244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.257202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.299327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.347259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.420357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 549.496479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.536158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.606516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.655785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.691674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.753710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 549.809801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.839973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.884955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.952237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.988628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.046405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 550.110509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.139705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.190168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 550.271847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.300848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.330328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 550.386293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.413807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.471887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 550.533392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.584634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.635278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 550.678359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.713703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.764700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 555.976648] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:51:15 (1761238275) [ 557.557255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg358-server [ 559.267565] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:51:18 (1761238278) [ 560.661576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg358-server [ 562.106365] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:51:21 (1761238281) [ 562.974786] Key type .llcrypt unregistered [ 562.977784] Key type ._llcrypt unregistered [ 567.706783] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 567.840873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 567.849401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 567.858864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 570.347338] Key type ._llcrypt registered [ 570.350037] Key type .llcrypt registered [ 570.540068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 570.551972] alg: No test for adler32 (adler32-zlib) [ 571.452872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.496170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 571.550663] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 572.094514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 572.104894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 573.782290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 573.800597] LNet: Removed LNI 10.1.2.3@tcp [ 573.919701] LNet: 34271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.925249] LNetError: 34271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.943172] LNet: Removed LNI 192.168.203.58@tcp [ 574.447342] Key type .llcrypt unregistered [ 574.448932] Key type ._llcrypt unregistered [ 582.810545] Key type ._llcrypt registered [ 582.811619] Key type .llcrypt registered [ 583.647939] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:51:43 (1761238303) [ 584.124645] Key type .llcrypt unregistered [ 584.126377] Key type ._llcrypt unregistered [ 587.907833] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 588.020145] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 588.022448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 588.024907] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 590.348989] Key type ._llcrypt registered [ 590.350269] Key type .llcrypt registered [ 590.506132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 590.517829] alg: No test for adler32 (adler32-zlib) [ 591.411300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.450122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.473990] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 592.009467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 592.029844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 593.785839] LNet: 35757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.789503] LNetError: 35757:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.794768] LNet: Removed LNI 192.168.203.58@tcp [ 593.802155] LNet: Removed LNI 10.1.2.3@tcp [ 594.243448] Key type .llcrypt unregistered [ 594.244614] Key type ._llcrypt unregistered [ 602.254731] Key type ._llcrypt registered [ 602.257419] Key type .llcrypt registered [ 603.357779] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:52:02 (1761238322) [ 603.835654] Key type .llcrypt unregistered [ 603.838054] Key type ._llcrypt unregistered [ 608.184863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 608.310869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 608.318594] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 608.320634] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 610.845795] Key type ._llcrypt registered [ 610.847839] Key type .llcrypt registered [ 611.021480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 611.047294] alg: No test for adler32 (adler32-zlib) [ 611.920412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.971919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 611.996549] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 613.628895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 613.654831] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 615.376101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 615.394450] LNet: Removed LNI 10.1.2.3@tcp [ 615.405611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 615.434947] LNet: Removed LNI 10.1.2.31@tcp [ 615.451538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 615.472730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 616.047308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 616.066608] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 616.849164] LNet: 37669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.856427] LNetError: 37669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 616.865085] LNet: Removed LNI 10.1.2.3@tcp [ 616.881067] LNet: Removed LNI 10.1.2.31@tcp [ 617.407483] Key type .llcrypt unregistered [ 617.409151] Key type ._llcrypt unregistered [ 626.924697] Key type ._llcrypt registered [ 626.929373] Key type .llcrypt registered [ 628.106884] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:52:27 (1761238347) [ 628.400394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.424217] alg: No test for adler32 (adler32-zlib) [ 629.314116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.367779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.384972] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 629.946277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.959794] LNet: 38614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.964635] LNetError: 38614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.979855] LNet: Removed LNI 192.168.203.58@tcp [ 629.999333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.052325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.58@tcp [ 630.073428] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 630.612959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.58@tcp [ 630.691515] LNet: Removed LNI 192.168.203.58@tcp [ 630.740308] LNet: 38746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.372476] Key type .llcrypt unregistered [ 631.374150] Key type ._llcrypt unregistered [ 639.494153] Key type ._llcrypt registered [ 639.497803] Key type .llcrypt registered [ 640.617135] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:52:40 (1761238360) [ 642.064375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg358-server [ 643.281733] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:52:43 (1761238363) [ 643.775357] Key type .llcrypt unregistered [ 643.779691] Key type ._llcrypt unregistered [ 648.108643] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 648.161239] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 648.172106] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 648.179794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 650.657958] Key type ._llcrypt registered [ 650.659562] Key type .llcrypt registered [ 650.806360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.820411] alg: No test for adler32 (adler32-zlib) [ 651.723848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.790340] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 652.398555] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 654.165409] LNet: 40570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.169604] LNetError: 40570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.173120] LNet: Removed LNI 192.168.203.58@tcp1 [ 654.184155] LNet: Removed LNI 10.1.2.3@tcp2 [ 654.696630] Key type .llcrypt unregistered [ 654.698316] Key type ._llcrypt unregistered [ 662.775731] Key type ._llcrypt registered [ 662.777246] Key type .llcrypt registered [ 663.803817] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:53:03 (1761238383) [ 664.148613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.169207] alg: No test for adler32 (adler32-zlib) [ 665.062270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.114668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.132451] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 665.956159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.972713] LNet: 41533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.978883] LNetError: 41533:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.989525] LNet: Removed LNI 192.168.203.58@tcp [ 666.012873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.067668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 666.085369] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 672.031273] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:53:11 (1761238391) [ 672.377012] LNet: 42125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.381584] LNetError: 42125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.397092] LNet: Removed LNI 192.168.203.58@tcp [ 672.988553] Key type .llcrypt unregistered [ 672.993930] Key type ._llcrypt unregistered [ 678.059083] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 678.228634] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.232138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 678.237580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.638535] Key type ._llcrypt registered [ 680.640343] Key type .llcrypt registered [ 680.860892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 680.879226] alg: No test for adler32 (adler32-zlib) [ 681.832620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.925721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.951625] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 682.514856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 682.526703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.374526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.390202] LNet: 42856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.395742] LNetError: 42856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.399909] LNet: Removed LNI 192.168.203.58@tcp [ 684.406981] LNet: Removed LNI 10.1.2.3@tcp [ 684.423294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.503832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 684.534479] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 684.540504] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 685.026874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.040853] LNet: 42884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.046270] LNetError: 42884:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.050151] LNetError: 42884:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 685.055426] LNet: Removed LNI 192.168.203.58@tcp [ 685.068412] LNet: Removed LNI 10.1.2.3@tcp [ 685.095315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.184677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 685.228267] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 685.240115] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 685.559086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 685.575789] LNet: Added LNI 192.168.203.58@tcp2 [8/256/0/180] [ 686.419123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.437779] LNet: 43043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.442645] LNetError: 43043:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.446523] LNetError: 43043:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 686.459045] LNet: Removed LNI 192.168.203.58@tcp [ 686.463257] LNet: Removed LNI 10.1.2.3@tcp [ 686.475364] LNet: Removed LNI 192.168.203.58@tcp2 [ 686.498653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.576407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 686.598546] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 686.603357] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.609216] LNet: Added LNI 192.168.203.58@tcp2 [8/256/0/180] [ 692.188218] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:53:31 (1761238411) [ 692.452653] LNet: 43642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.466353] LNetError: 43642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.474652] LNetError: 43642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 692.496254] LNet: Removed LNI 192.168.203.58@tcp [ 692.500382] LNet: Removed LNI 10.1.2.3@tcp [ 692.515508] LNet: Removed LNI 192.168.203.58@tcp2 [ 693.016101] Key type .llcrypt unregistered [ 693.017718] Key type ._llcrypt unregistered [ 698.073994] Key type ._llcrypt registered [ 698.075536] Key type .llcrypt registered [ 698.321233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 698.337576] alg: No test for adler32 (adler32-zlib) [ 699.335921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.384299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.434603] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 700.305397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.321841] LNet: 44028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.326753] LNetError: 44028:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.338608] LNet: Removed LNI 192.168.203.58@tcp [ 700.357971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.424871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 700.450821] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 700.811932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.835158] LNet: 44058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.841342] LNetError: 44058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.853666] LNet: Removed LNI 192.168.203.58@tcp [ 700.867935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.924557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 700.940862] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 701.459577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.491905] LNet: Removed LNI 192.168.203.58@tcp [ 701.507723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.575134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 701.600566] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 702.070879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.180057] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.183453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.191295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 704.532996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.554524] LNet: 44160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.558471] LNet: 44160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 704.574476] LNetError: 44160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.579448] LNetError: 44160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 704.609991] LNet: Removed LNI 192.168.203.58@tcp [ 704.635871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.717985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 704.774938] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 710.186904] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:53:49 (1761238429) [ 710.438962] LNet: 44756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.851386] LNetError: 44756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.874895] LNet: Removed LNI 192.168.203.58@tcp [ 711.368972] Key type .llcrypt unregistered [ 711.370621] Key type ._llcrypt unregistered [ 717.237277] Key type ._llcrypt registered [ 717.238823] Key type .llcrypt registered [ 717.439202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.454480] alg: No test for adler32 (adler32-zlib) [ 718.385115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.435800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.489953] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 719.320259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.344214] LNet: 45148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.359361] LNetError: 45148:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.382949] LNet: Removed LNI 192.168.203.58@tcp [ 719.399597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.490423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 719.531242] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 725.519057] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:54:05 (1761238445) [ 725.680769] LNet: 45738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.692709] LNetError: 45738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.734980] LNet: Removed LNI 192.168.203.58@tcp [ 726.280474] Key type .llcrypt unregistered [ 726.282189] Key type ._llcrypt unregistered [ 730.757063] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 730.934270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 730.938875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 730.953640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.391433] Key type ._llcrypt registered [ 733.396753] Key type .llcrypt registered [ 733.542596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.555232] alg: No test for adler32 (adler32-zlib) [ 734.448794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.488986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.502434] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 735.261816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.280742] LNet: 46183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.294346] LNetError: 46183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.337023] LNet: Removed LNI 192.168.203.58@tcp [ 735.366130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.487371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 735.504483] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 740.660591] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:54:20 (1761238460) [ 740.808555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.819743] LNet: 46782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.822812] LNetError: 46782:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.832750] LNet: Removed LNI 192.168.203.58@tcp [ 740.848057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.923203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 745.205497] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:54:25 (1761238465) [ 745.335787] LNet: 47359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.806811] Key type .llcrypt unregistered [ 745.808702] Key type ._llcrypt unregistered [ 748.949742] Key type ._llcrypt registered [ 748.951927] Key type .llcrypt registered [ 749.082771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.097471] alg: No test for adler32 (adler32-zlib) [ 749.979795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.021476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.042904] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 750.775994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.793841] LNet: 47753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.804859] LNetError: 47753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.819822] LNet: Removed LNI 192.168.203.58@tcp [ 750.835727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.878217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 750.900982] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 757.509605] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:54:36 (1761238476) [ 757.630853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.662455] LNet: 48342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.681057] LNetError: 48342:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.723649] LNet: Removed LNI 192.168.203.58@tcp [ 757.771665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.871182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 763.838629] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:54:43 (1761238483) [ 764.295586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.317232] LNet: 48926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.349367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.430780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.459662] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 765.140359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.58 [ 765.162363] LNet: Removed LNI 192.168.203.58@tcp [ 770.386557] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:54:50 (1761238490) [ 770.599321] LNet: 49625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.603056] LNetError: 49625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.626074] LNet: Removed LNI 192.168.203.58@tcp [ 771.205702] Key type .llcrypt unregistered [ 771.207508] Key type ._llcrypt unregistered [ 776.955184] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.163505] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.172264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.176734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.743968] Key type ._llcrypt registered [ 779.745675] Key type .llcrypt registered [ 779.907368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.919412] alg: No test for adler32 (adler32-zlib) [ 781.157903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.646377] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:55:06 (1761238506) [ 786.866820] LNet: 50525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.876842] LNetError: 50525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 786.907376] LNet: Removed LNI 10.1.2.3@tcp [ 787.473593] Key type .llcrypt unregistered [ 787.475054] Key type ._llcrypt unregistered [ 793.317445] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 793.533595] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 793.547212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 793.559833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 796.193059] Key type ._llcrypt registered [ 796.196957] Key type .llcrypt registered [ 796.362833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.389578] alg: No test for adler32 (adler32-zlib) [ 797.320217] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.309209] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:55:22 (1761238522) [ 802.614114] LNet: 51415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.627608] LNetError: 51415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.636753] LNet: Removed LNI 10.1.2.3@tcp [ 803.128483] Key type .llcrypt unregistered [ 803.130311] Key type ._llcrypt unregistered [ 807.878727] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 808.061720] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.063760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 808.074545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.581211] Key type ._llcrypt registered [ 809.585118] Key type .llcrypt registered [ 809.711017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.722902] alg: No test for adler32 (adler32-zlib) [ 810.616346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.018194] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:55:34 (1761238534) [ 815.143585] LNet: 52290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.156788] LNetError: 52290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 815.183281] LNet: Removed LNI 10.1.2.3@tcp [ 815.870890] Key type .llcrypt unregistered [ 815.877031] Key type ._llcrypt unregistered [ 820.528362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.746689] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.755657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.772099] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.510436] Key type ._llcrypt registered [ 823.512033] Key type .llcrypt registered [ 823.698963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.716567] alg: No test for adler32 (adler32-zlib) [ 824.588224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.643245] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 831.360425] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:55:51 (1761238551) [ 831.816124] LNet: 53519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.823261] LNetError: 53519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.840515] LNet: Removed LNI 10.1.2.3@tcp [ 832.337550] Key type .llcrypt unregistered [ 832.339185] Key type ._llcrypt unregistered [ 836.761674] Key type ._llcrypt registered [ 836.763019] Key type .llcrypt registered [ 836.929798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 836.943784] alg: No test for adler32 (adler32-zlib) [ 837.845697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 837.873511] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 843.883855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 844.499385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_interrupt -r 1 [ 844.525563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_interrupt -r 1 [ 844.555575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 844.574774] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 844.819899] LNet: 53945:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 844.896106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 844.926160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 844.952852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 845.354670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_dropped -r 1 [ 845.384771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_dropped -r 1 [ 845.425839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 845.447577] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 845.754936] LNet: 54042:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 845.818201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 845.852614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 845.896277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 846.280166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_aborted -r 1 [ 846.313300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_aborted -r 1 [ 846.347392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 846.652243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 846.673119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 846.703791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 847.051112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_no_route -r 1 [ 847.079172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_no_route -r 1 [ 847.107462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 847.133678] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 847.142057] LNet: Skipped 1 previous similar message [ 847.347118] LNet: 54236:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 847.360503] LNet: 54236:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 847.404545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 847.424452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 847.443076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 847.794434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_timeout -r 1 [ 847.822628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_timeout -r 1 [ 847.858260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 848.157804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 848.179884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 848.207919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 848.600823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_error -r 1 [ 848.634204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_error -r 1 [ 848.670727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 848.982778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 849.013224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 849.037284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 849.261601] LNet: 54466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.269536] LNetError: 54466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.287393] LNet: Removed LNI 192.168.203.58@tcp [ 849.560751] Key type .llcrypt unregistered [ 849.564717] Key type ._llcrypt unregistered [ 862.698293] Key type ._llcrypt registered [ 862.699748] Key type .llcrypt registered [ 864.008189] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:56:23 (1761238583) [ 865.168468] Key type .llcrypt unregistered [ 865.174620] Key type ._llcrypt unregistered [ 871.641969] Key type ._llcrypt registered [ 871.643542] Key type .llcrypt registered [ 871.961399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.971761] alg: No test for adler32 (adler32-zlib) [ 872.930832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 872.956386] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 879.879811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 888.170799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.220910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 888.252382] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 889.624172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_interrupt -r 1 [ 889.669520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_interrupt -r 1 [ 889.699366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_interrupt -r 1 [ 889.726240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_interrupt -r 1 [ 889.768910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_interrupt -r 1 [ 889.788786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_interrupt -r 1 [ 889.817692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_interrupt -r 1 [ 889.849411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_interrupt -r 1 [ 889.880576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 889.902251] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 889.911759] LNet: 1 local NIs in recovery (showing 1): 192.168.203.58@tcp [ 890.226895] LNet: 55915:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 890.327726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 890.360258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 890.392336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 890.963460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_dropped -r 1 [ 890.993478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_dropped -r 1 [ 891.016172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_dropped -r 1 [ 891.036296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_dropped -r 1 [ 891.060521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_dropped -r 1 [ 891.088460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_dropped -r 1 [ 891.136154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_dropped -r 1 [ 891.168780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_dropped -r 1 [ 891.197743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 891.221112] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 891.229542] LNet: Skipped 2 previous similar messages [ 891.508360] LNet: 56025:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 891.567473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 891.613945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 891.648946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 892.072451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_aborted -r 1 [ 892.104518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_aborted -r 1 [ 892.133378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_aborted -r 1 [ 892.173062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_aborted -r 1 [ 892.195740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_aborted -r 1 [ 892.227453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_aborted -r 1 [ 892.278989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_aborted -r 1 [ 892.305945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_aborted -r 1 [ 892.342466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 892.368411] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 892.372169] LNet: Skipped 2 previous similar messages [ 892.626484] LNet: 56134:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 892.685747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 892.725174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 892.758502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 893.154235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_no_route -r 1 [ 893.189098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_no_route -r 1 [ 893.225718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_no_route -r 1 [ 893.259622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_no_route -r 1 [ 893.291447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_no_route -r 1 [ 893.330375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_no_route -r 1 [ 893.376890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_no_route -r 1 [ 893.421805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_no_route -r 1 [ 893.463183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 893.785976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 893.807487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 893.835607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 894.316174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_timeout -r 1 [ 894.352443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_timeout -r 1 [ 894.395872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_timeout -r 1 [ 894.426641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_timeout -r 1 [ 894.462903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_timeout -r 1 [ 894.491930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_timeout -r 1 [ 894.516859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_timeout -r 1 [ 894.545191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_timeout -r 1 [ 894.569647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 894.590205] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 894.593845] LNet: Skipped 5 previous similar messages [ 894.892899] LNet: 56352:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 894.896491] LNet: 56352:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 894.957780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 895.002807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 895.039741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 895.583653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_error -r 1 [ 895.627375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_error -r 1 [ 895.674456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_error -r 1 [ 895.698896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_error -r 1 [ 895.722683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_error -r 1 [ 895.753103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_error -r 1 [ 895.773589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_error -r 1 [ 895.832965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_error -r 1 [ 895.865506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 896.242766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 896.277971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 896.318124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 900.349734] LNet: 56532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.354901] LNetError: 56532:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.359475] LNet: Removed LNI 192.168.203.58@tcp [ 900.375148] LNet: Removed LNI 192.168.203.58@tcp1 [ 900.883586] Key type .llcrypt unregistered [ 900.888222] Key type ._llcrypt unregistered [ 914.966321] Key type ._llcrypt registered [ 914.968455] Key type .llcrypt registered [ 916.145109] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:57:15 (1761238635) [ 917.036574] Key type .llcrypt unregistered [ 917.037899] Key type ._llcrypt unregistered [ 921.795310] Key type ._llcrypt registered [ 921.797850] Key type .llcrypt registered [ 922.025905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.045211] alg: No test for adler32 (adler32-zlib) [ 922.918912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 922.948093] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 929.674928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 930.448926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_dropped -r 1 [ 930.482558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_dropped -r 1 [ 930.524135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 930.548592] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 930.860570] LNet: 57768:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 930.918972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.941702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.985352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 931.617658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_error -r 1 [ 931.648721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_error -r 1 [ 931.713690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 931.739665] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 932.051705] LNet: 57866:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 932.107511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.131837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.181589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 932.659755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_timeout -r 1 [ 932.692706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_timeout -r 1 [ 932.731974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 932.752590] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 933.073533] LNet: 57963:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 933.152585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 933.205680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 933.251183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 933.478261] LNet: 57999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.488285] LNetError: 57999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.498613] LNet: Removed LNI 192.168.203.58@tcp [ 933.920551] Key type .llcrypt unregistered [ 933.922433] Key type ._llcrypt unregistered [ 946.397488] Key type ._llcrypt registered [ 946.398956] Key type .llcrypt registered [ 947.248411] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:57:47 (1761238667) [ 947.943307] Key type .llcrypt unregistered [ 947.945104] Key type ._llcrypt unregistered [ 952.754611] Key type ._llcrypt registered [ 952.759348] Key type .llcrypt registered [ 953.111558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 953.135249] alg: No test for adler32 (adler32-zlib) [ 954.050202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 954.074665] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 961.084604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 967.809354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.902782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 967.959808] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 969.355737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_dropped -r 1 [ 969.416646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e remote_dropped -r 1 [ 969.479566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_dropped -r 1 [ 969.557043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e remote_dropped -r 1 [ 969.605850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e remote_dropped -r 1 [ 969.648484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e remote_dropped -r 1 [ 969.706256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e remote_dropped -r 1 [ 969.734603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e remote_dropped -r 1 [ 969.766587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 969.781261] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 969.788759] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.158@tcp [ 970.097638] LNet: 59447:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.161929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.187336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 970.223448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.255555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.721131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_error -r 1 [ 970.762507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e remote_error -r 1 [ 970.790077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_error -r 1 [ 970.859308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e remote_error -r 1 [ 970.924590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e remote_error -r 1 [ 970.973329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e remote_error -r 1 [ 971.008786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e remote_error -r 1 [ 971.049427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e remote_error -r 1 [ 971.085520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 971.103114] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 971.106790] LNet: Skipped 2 previous similar messages [ 971.428539] LNet: 59558:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 971.480988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.520283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.537957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 971.561323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 972.056523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_timeout -r 1 [ 972.105822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e remote_timeout -r 1 [ 972.136537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_timeout -r 1 [ 972.168441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e remote_timeout -r 1 [ 972.202470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e remote_timeout -r 1 [ 972.246581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e remote_timeout -r 1 [ 972.279157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e remote_timeout -r 1 [ 972.322199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e remote_timeout -r 1 [ 972.356307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.158@tcp [ 972.380353] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 972.616959] LNet: 59669:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 972.687430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 972.726613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 972.754462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 972.768762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.325305] LNet: 59742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.332906] LNetError: 59742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.338504] LNet: Removed LNI 192.168.203.58@tcp [ 977.385525] LNet: Removed LNI 192.168.203.58@tcp1 [ 977.895514] Key type .llcrypt unregistered [ 977.897183] Key type ._llcrypt unregistered [ 991.605428] Key type ._llcrypt registered [ 991.606986] Key type .llcrypt registered [ 992.993967] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:58:32 (1761238712) [ 993.392280] Key type .llcrypt unregistered [ 993.393615] Key type ._llcrypt unregistered [ 998.931349] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.973251] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 998.975667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.093587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.734045] Key type ._llcrypt registered [ 1001.736186] Key type .llcrypt registered [ 1001.881409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.891748] alg: No test for adler32 (adler32-zlib) [ 1002.851441] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1002.882108] LNet: 60823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.894508] LNetError: 60823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.912621] LNet: Removed LNI 192.168.203.58@tcp [ 1003.547412] Key type .llcrypt unregistered [ 1003.549034] Key type ._llcrypt unregistered [ 1007.949652] Key type ._llcrypt registered [ 1007.951351] Key type .llcrypt registered [ 1008.118033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.126683] alg: No test for adler32 (adler32-zlib) [ 1008.985686] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1008.988104] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1009.015219] LNet: 61092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.019402] LNetError: 61092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.023104] LNet: Removed LNI 192.168.203.58@tcp [ 1009.033643] LNet: Removed LNI 10.1.2.3@tcp1 [ 1009.605415] Key type .llcrypt unregistered [ 1009.606657] Key type ._llcrypt unregistered [ 1014.098075] Key type ._llcrypt registered [ 1014.099329] Key type .llcrypt registered [ 1014.327442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.344596] alg: No test for adler32 (adler32-zlib) [ 1015.233846] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1015.237208] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1015.260517] LNet: 61360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.264705] LNetError: 61360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.268187] LNet: Removed LNI 192.168.203.58@tcp [ 1015.279621] LNet: Removed LNI 10.1.2.3@tcp [ 1015.845199] Key type .llcrypt unregistered [ 1015.846886] Key type ._llcrypt unregistered [ 1020.404377] Key type ._llcrypt registered [ 1020.405986] Key type .llcrypt registered [ 1020.551241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.567339] alg: No test for adler32 (adler32-zlib) [ 1021.533167] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1021.537108] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1021.577388] LNet: 61630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.586422] LNetError: 61630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.589635] LNet: Removed LNI 192.168.203.58@tcp [ 1021.610112] LNet: Removed LNI 10.1.2.3@tcp [ 1022.136674] Key type .llcrypt unregistered [ 1022.139597] Key type ._llcrypt unregistered [ 1025.738390] Key type ._llcrypt registered [ 1025.739950] Key type .llcrypt registered [ 1025.894567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.906425] alg: No test for adler32 (adler32-zlib) [ 1026.794853] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1026.796498] LNetError: 61890:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1026.799802] LNetError: Error -100 starting up LNI tcp [ 1026.812850] LNet: Removed LNI 192.168.203.58@tcp [ 1027.290589] Key type .llcrypt unregistered [ 1027.292666] Key type ._llcrypt unregistered [ 1034.482305] Key type ._llcrypt registered [ 1034.484256] Key type .llcrypt registered [ 1035.448868] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:59:15 (1761238755) [ 1036.089529] Key type .llcrypt unregistered [ 1036.092685] Key type ._llcrypt unregistered [ 1039.963351] Key type ._llcrypt registered [ 1039.964726] Key type .llcrypt registered [ 1040.131366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.142454] alg: No test for adler32 (adler32-zlib) [ 1041.049926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1041.074600] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1046.574193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1047.089927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e network_timeout -r 1 [ 1047.135568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e network_timeout -r 1 [ 1047.167387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1047.189091] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1047.506476] LNet: 63150:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.557358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.585111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.729611] LNet: 63184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.733988] LNetError: 63184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.745916] LNet: Removed LNI 192.168.203.58@tcp [ 1048.004416] Key type .llcrypt unregistered [ 1048.006912] Key type ._llcrypt unregistered [ 1059.022519] Key type ._llcrypt registered [ 1059.024258] Key type .llcrypt registered [ 1059.227797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.238810] alg: No test for adler32 (adler32-zlib) [ 1060.184921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1060.215744] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1066.127966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1072.396074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.433134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1072.456177] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1073.770390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e network_timeout -r 1 [ 1073.804420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e network_timeout -r 1 [ 1073.831684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e network_timeout -r 1 [ 1073.874652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e network_timeout -r 1 [ 1073.930891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e network_timeout -r 1 [ 1073.965783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e network_timeout -r 1 [ 1074.020565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e network_timeout -r 1 [ 1074.052668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e network_timeout -r 1 [ 1074.082537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1074.101516] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1074.350809] LNet: 64044:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.396859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.426533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.724672] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1077.952625] LNet: 64113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.964159] LNetError: 64113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.971196] LNet: Removed LNI 192.168.203.58@tcp [ 1077.983941] LNet: Removed LNI 192.168.203.58@tcp1 [ 1078.249753] Key type .llcrypt unregistered [ 1078.255632] Key type ._llcrypt unregistered [ 1089.906613] Key type ._llcrypt registered [ 1089.908268] Key type .llcrypt registered [ 1090.891251] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:00:10 (1761238810) [ 1091.146536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1091.158446] alg: No test for adler32 (adler32-zlib) [ 1092.019496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.068631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1092.082658] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1092.646085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1092.666517] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1093.319607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1093.388152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1093.460223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1093.485528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1093.515385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1093.543125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1093.576274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.58@tcp [ 1093.592839] LNetError: 65203:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.203.58@tcp to recovery [ 1094.111283] LNet: 1 local NIs in recovery (showing 1): 192.168.203.58@tcp [ 1094.112606] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1096.159423] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1097.026172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1100.256057] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1100.418602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1100.452626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.496735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.520529] LNet: 65310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.532567] LNetError: 65310:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.547967] LNet: Removed LNI 192.168.203.58@tcp [ 1100.562672] LNet: Removed LNI 192.168.203.58@tcp1 [ 1100.588490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.644745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1100.664930] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1101.196171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1101.206920] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1101.897143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1101.926056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.964819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1102.001035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1102.037932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1102.068209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1102.093079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1102.131364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.58@tcp [ 1102.149698] LNetError: 65577:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.203.58@tcp to recovery [ 1102.688563] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1105.529352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1108.831654] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1108.835136] LNet: Skipped 1 previous similar message [ 1113.162272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1113.200835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.231266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.320909] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:00:38 (1761238838) [ 1118.378357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1118.399476] LNet: 66286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.404519] LNetError: 66286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.409160] LNetError: 66286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1118.418129] LNet: Removed LNI 192.168.203.58@tcp [ 1118.426452] LNet: Removed LNI 192.168.203.58@tcp1 [ 1118.450248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.508192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1118.539366] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1118.988251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1119.000701] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1119.541475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1119.584438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1119.606434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1119.627331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1119.655607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1119.685883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1119.712432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.58@tcp --health 0 [ 1119.773063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1120.543215] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.58@tcp [ 1120.545040] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1120.545045] LNet: Skipped 1 previous similar message [ 1135.811448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1135.913523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1135.938536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.961079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.58@tcp --health 500 [ 1136.047907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1142.534349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1142.606858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1142.636318] LNet: 66799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.645470] LNetError: 66799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.652355] LNetError: 66799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1142.661183] LNet: Removed LNI 192.168.203.58@tcp [ 1142.674679] LNet: Removed LNI 192.168.203.58@tcp1 [ 1142.698447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.761760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1142.800076] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1143.421431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1143.432706] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1144.075955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1144.103044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.147794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1144.177254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1144.213744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1144.247268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1144.284291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1144.321274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.58@tcp --health 0 [ 1145.825584] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1145.831670] LNet: Skipped 3 previous similar messages [ 1146.660109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1155.607782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1155.644978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.686475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.126575] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:01:21 (1761238881) [ 1162.546713] LNet: 67840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.553875] LNetError: 67840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.558249] LNetError: 67840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1162.562865] LNet: Removed LNI 192.168.203.58@tcp [ 1162.573862] LNet: Removed LNI 192.168.203.58@tcp1 [ 1163.069988] Key type .llcrypt unregistered [ 1163.071945] Key type ._llcrypt unregistered [ 1167.661062] Key type ._llcrypt registered [ 1167.663454] Key type .llcrypt registered [ 1167.850619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.862187] alg: No test for adler32 (adler32-zlib) [ 1168.749829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1168.781782] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1175.013460] Lustre: DEBUG MARKER: Initial discovery [ 1175.134761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.158@tcp [ 1176.696832] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1176.837237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1176.864747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.158@tcp [ 1176.880870] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1176.899825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.923873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.158@tcp -r 1 -m GET -l 3 [ 1176.949841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.158@tcp -r 1 -m GET -e local_error [ 1176.972696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.158@tcp -r 1 -m PUT -l 6 [ 1177.922720] Lustre: DEBUG MARKER: Force oleg358-client.virtnet to discover 192.168.203.158@tcp (in background) [ 1178.007488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.158@tcp [ 1178.919940] Lustre: DEBUG MARKER: Force oleg358-server to discover 192.168.203.58@tcp [ 1180.678344] Lustre: DEBUG MARKER: Wait for 68593 [ 1181.091940] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1182.072513] Lustre: DEBUG MARKER: Finished wait on 68593 [ 1182.184739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1182.205723] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1182.210769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.343390] LNet: 69045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.350396] LNetError: 69045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.365222] LNet: Removed LNI 192.168.203.58@tcp [ 1182.605357] Key type .llcrypt unregistered [ 1182.608479] Key type ._llcrypt unregistered [ 1194.531736] Key type ._llcrypt registered [ 1194.534174] Key type .llcrypt registered [ 1195.659348] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:01:55 (1761238915) [ 1196.082550] Key type .llcrypt unregistered [ 1196.085798] Key type ._llcrypt unregistered [ 1200.230756] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1200.311898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1200.313833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1200.328026] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1202.630277] Key type ._llcrypt registered [ 1202.631806] Key type .llcrypt registered [ 1202.753448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.762553] alg: No test for adler32 (adler32-zlib) [ 1203.616221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1203.661553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1203.681806] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1204.167976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1204.187278] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1211.178976] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:02:10 (1761238930) [ 1211.256255] LNet: 71093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.261565] LNetError: 71093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.266231] LNet: Removed LNI 192.168.203.58@tcp [ 1211.276194] LNet: Removed LNI 10.1.2.3@tcp [ 1211.815366] Key type .llcrypt unregistered [ 1211.824448] Key type ._llcrypt unregistered [ 1216.519497] Key type ._llcrypt registered [ 1216.523745] Key type .llcrypt registered [ 1216.695558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.721490] alg: No test for adler32 (adler32-zlib) [ 1217.624436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.670454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1217.697261] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1218.229284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1218.248227] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1218.898432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.58@tcp [ 1219.936617] LNetError: Unexpected error -22 connecting to 192.168.203.58@tcp at host 192.168.203.58:988 [ 1223.222751] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:02:23 (1761238943) [ 1223.289680] LNet: 72216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.293887] LNetError: 72216:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.300341] LNet: Removed LNI 192.168.203.58@tcp [ 1223.311525] LNet: Removed LNI 10.1.2.3@tcp [ 1223.825435] Key type .llcrypt unregistered [ 1223.826654] Key type ._llcrypt unregistered [ 1227.741630] Key type ._llcrypt registered [ 1227.743572] Key type .llcrypt registered [ 1227.907717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.923877] alg: No test for adler32 (adler32-zlib) [ 1228.771094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.815578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1228.848928] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1229.386791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1229.397804] LNet: Added LNI 192.168.203.58@tcp2 [8/256/0/180] [ 1229.913197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.58@tcp1 --nid 192.168.203.58@tcp2 [ 1232.681924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp1 192.168.203.58@tcp2 [ 1232.691774] LNetError: 73047:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.58@tcp1 to 192.168.203.58@tcp2 - Route table may be misconfigured [ 1232.707195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp2 192.168.203.58@tcp1 [ 1236.503268] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:02:36 (1761238956) [ 1236.614954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1236.645045] LNet: 73621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.650181] LNetError: 73621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.656928] LNet: Removed LNI 192.168.203.58@tcp1 [ 1237.672252] LNet: Removed LNI 192.168.203.58@tcp2 [ 1237.689614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.728728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1237.746770] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1238.175269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.190165] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1238.720507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1238.746183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.58@tcp [ 1238.781541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp1 192.168.203.58@tcp1 [ 1238.806992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.58@tcp -d 192.168.203.58@tcp -e network_timeout [ 1238.833606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -e network_timeout [ 1238.859647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -e network_timeout [ 1238.886099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -e network_timeout [ 1238.907677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1238.921752] LNet: There was an unexpected network error while writing to 192.168.203.58: rc = -22 [ 1238.943157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1239.011636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1239.071279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1239.775248] LNet: 1 local NIs in recovery (showing 1): 192.168.203.58@tcp [ 1243.268881] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:02:43 (1761238963) [ 1243.371597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.386513] LNet: 74474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.390991] LNetError: 74474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.395466] LNetError: 74474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1243.399573] LNet: Removed LNI 192.168.203.58@tcp [ 1244.453769] LNet: Removed LNI 192.168.203.58@tcp1 [ 1244.484910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.546881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1244.713629] LNet: 74512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.966835] Key type .llcrypt unregistered [ 1244.968600] Key type ._llcrypt unregistered [ 1251.399191] Key type ._llcrypt registered [ 1251.400955] Key type .llcrypt registered [ 1252.332152] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:02:52 (1761238972) [ 1252.659672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1252.673921] alg: No test for adler32 (adler32-zlib) [ 1253.579108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1254.660791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1255.768553] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:02:55 (1761238975) [ 1255.961700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1255.978536] LNet: 75605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.000493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.033339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1256.053966] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1256.548584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1256.562042] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1257.143986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1257.175356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp1 [ 1257.201954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp1 [ 1261.541854] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:03:01 (1761238981) [ 1262.867568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg358-server [ 1264.348763] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:03:04 (1761238984) [ 1265.985884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg358-server [ 1267.304547] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:03:07 (1761238987) [ 1268.738509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg358-server [ 1270.053775] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:03:09 (1761238989) [ 1271.415736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg358-server [ 1272.603490] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:03:12 (1761238992) [ 1273.705501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg358-server [ 1274.858669] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:03:14 (1761238994) [ 1276.094338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg358-server [ 1277.426797] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:03:17 (1761238997) [ 1278.868393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg358-server [ 1280.083145] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:03:19 (1761238999) [ 1281.430987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg358-server [ 1282.815676] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:03:22 (1761239002) [ 1284.369826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg358-server [ 1285.727791] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:03:25 (1761239005) [ 1285.828605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1285.851618] LNet: 79469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.855175] LNetError: 79469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.861567] LNet: Removed LNI 192.168.203.58@tcp [ 1286.892614] LNet: Removed LNI 192.168.203.58@tcp1 [ 1286.913575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.988378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1287.029750] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1287.631485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1287.694210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1287.847720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.864425] LNet: 79618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.873633] LNetError: 79618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.884335] LNetError: 79618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1287.898881] LNet: Removed LNI 192.168.203.58@tcp [ 1287.919769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.972203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1288.005438] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1288.679125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1288.889942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.910816] LNetError: 79760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.928639] LNet: Removed LNI 192.168.203.58@tcp [ 1288.960401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.010906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1289.042897] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1289.580692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1289.647704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1290.873996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1290.893241] LNet: 79913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.897833] LNet: 79913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1291.943344] LNet: Removed LNI 192.168.203.58@tcp [ 1291.963967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.008430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1292.037228] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1292.650441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1292.831288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1292.862398] LNetError: 80056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.874529] LNetError: 80056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1292.892798] LNet: Removed LNI 192.168.203.58@tcp [ 1292.916446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.969692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1293.005919] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1293.662966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1293.756489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1293.979803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1295.015925] LNet: Removed LNI 192.168.203.58@tcp [ 1295.030616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.065943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1295.098823] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1295.607924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1295.772724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1295.789524] LNet: 80342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.793415] LNet: 80342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1296.872438] LNet: Removed LNI 192.168.203.58@tcp [ 1296.897738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.941318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1296.958375] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1297.438348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1297.484865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1297.637902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.667229] LNetError: 80486:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.670863] LNetError: 80486:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1297.685111] LNet: Removed LNI 192.168.203.58@tcp [ 1297.707774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.744629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1297.759502] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1298.292669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1299.543782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.582508] LNet: Removed LNI 192.168.203.58@tcp [ 1300.606951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.657172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1300.686917] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1301.266811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1301.330629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1301.647399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.694217] LNet: Removed LNI 192.168.203.58@tcp [ 1302.716450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.774779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1302.798258] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1303.346927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1303.571639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1304.612920] LNet: Removed LNI 192.168.203.58@tcp [ 1304.641083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.694198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.718699] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1305.322554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1305.400735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1305.604380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1305.624916] LNet: 81069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.628240] LNet: 81069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1306.661961] LNet: Removed LNI 192.168.203.58@tcp [ 1306.695350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.755477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1306.773824] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1307.372278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1307.564037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1307.583771] LNetError: 81211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.590704] LNetError: 81211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1307.618245] LNet: Removed LNI 192.168.203.58@tcp [ 1307.634164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1307.665154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1307.684653] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1308.175864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1308.212845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1308.243035] LNet: Removed LNI 192.168.203.58@tcp [ 1308.263089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.304214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.327853] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1308.872221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1313.549512] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:03:53 (1761239033) [ 1313.682718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1313.719231] LNet: Removed LNI 192.168.203.58@tcp [ 1313.747293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.802292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1313.822707] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/180] [ 1314.716190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.740686] LNet: Removed LNI 192.168.203.58@tcp231 [ 1314.792189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.860545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1314.887942] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/180] [ 1314.895314] LNet: 82185:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.198892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1315.219716] LNet: Removed LNI 192.168.203.58@tcp231 [ 1315.236292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1315.256827] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/180] [ 1316.107790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.150730] LNet: Removed LNI 192.168.203.58@tcp231 [ 1316.170611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.218432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1316.238962] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/0] [ 1316.249912] LNet: 82335:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.627816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1316.664822] LNet: Removed LNI 192.168.203.58@tcp231 [ 1316.683667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1316.704581] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/0] [ 1317.495539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1317.514675] LNet: Removed LNI 192.168.203.58@tcp231 [ 1317.530266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.585952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1317.622504] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/60] [ 1317.632321] LNet: 82483:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.933795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1317.957726] LNet: Removed LNI 192.168.203.58@tcp231 [ 1317.974459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1317.991846] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/60] [ 1318.863624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1318.895927] LNet: Removed LNI 192.168.203.58@tcp231 [ 1318.923849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.989568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1319.016029] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/180] [ 1319.284301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1319.322162] LNet: Removed LNI 192.168.203.58@tcp231 [ 1319.344108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1319.380803] LNet: Added LNI 192.168.203.58@tcp231 [8/256/0/180] [ 1325.369119] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:04:04 (1761239044) [ 1325.436093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.453588] LNet: 83335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.459920] LNet: 83335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1325.464051] LNetError: 83335:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.472348] LNetError: 83335:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1325.492309] LNet: Removed LNI 192.168.203.58@tcp231 [ 1325.511258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.552125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1325.567483] LNet: Added LNI 192.168.203.58@tcp232 [8/256/0/180] [ 1326.140873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1326.177692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1326.202572] LNet: Removed LNI 192.168.203.58@tcp232 [ 1326.226497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1326.268807] LNet: Added LNI 192.168.203.58@tcp232 [8/256/0/180] [ 1326.833934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1326.862842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1326.892845] LNet: Removed LNI 192.168.203.58@tcp232 [ 1326.906936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1326.932835] LNet: Added LNI 192.168.203.58@tcp232 [8/256/0/180] [ 1327.566442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1327.598343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1327.624555] LNet: Removed LNI 192.168.203.58@tcp232 [ 1327.636409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1327.657809] LNet: Added LNI 192.168.203.58@tcp232 [8/256/0/180] [ 1328.178397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1328.232274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1328.262940] LNet: Removed LNI 192.168.203.58@tcp232 [ 1328.283153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1333.582151] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:04:13 (1761239053) [ 1334.643217] Key type .llcrypt unregistered [ 1334.651359] Key type ._llcrypt unregistered [ 1339.994423] Key type ._llcrypt registered [ 1339.996016] Key type .llcrypt registered [ 1340.177797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.187180] alg: No test for adler32 (adler32-zlib) [ 1341.121139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1341.142293] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1347.067674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1354.112785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.165440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.207787] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1355.650927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_interrupt -r 2 [ 1355.677680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_interrupt -r 2 [ 1355.708213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_interrupt -r 2 [ 1355.733887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_interrupt -r 2 [ 1355.770166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_interrupt -r 2 [ 1355.807590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_interrupt -r 2 [ 1355.830940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_interrupt -r 2 [ 1355.866685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_interrupt -r 2 [ 1355.895173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.158@tcp [ 1355.917815] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1355.923129] LNet: 1 local NIs in recovery (showing 1): 192.168.203.58@tcp [ 1356.176050] LNet: 85102:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1356.226904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.246208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.625093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_dropped -r 2 [ 1356.656096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_dropped -r 2 [ 1356.678802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_dropped -r 2 [ 1356.706513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_dropped -r 2 [ 1356.731691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_dropped -r 2 [ 1356.753235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_dropped -r 2 [ 1356.776113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_dropped -r 2 [ 1356.796246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_dropped -r 2 [ 1356.824244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.158@tcp [ 1356.845057] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1357.137871] LNet: 85211:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.233946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.261362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1357.691672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_aborted -r 2 [ 1357.719252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_aborted -r 2 [ 1357.755837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_aborted -r 2 [ 1357.786406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_aborted -r 2 [ 1357.828061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_aborted -r 2 [ 1357.864799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_aborted -r 2 [ 1357.894352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_aborted -r 2 [ 1357.920722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_aborted -r 2 [ 1357.948434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.158@tcp [ 1357.971796] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1357.982289] LNet: Skipped 1 previous similar message [ 1358.271650] LNet: 85319:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1358.361360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1358.392223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.728042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_no_route -r 2 [ 1358.769290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_no_route -r 2 [ 1358.811343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_no_route -r 2 [ 1358.849618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_no_route -r 2 [ 1358.885169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_no_route -r 2 [ 1358.918615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_no_route -r 2 [ 1358.950332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_no_route -r 2 [ 1358.995099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_no_route -r 2 [ 1359.034330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.158@tcp [ 1359.402367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.435625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.940982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e local_timeout -r 2 [ 1359.975760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e local_timeout -r 2 [ 1359.999188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e local_timeout -r 2 [ 1360.027550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e local_timeout -r 2 [ 1360.059711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e local_timeout -r 2 [ 1360.094695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e local_timeout -r 2 [ 1360.132737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e local_timeout -r 2 [ 1360.160937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e local_timeout -r 2 [ 1360.200174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.158@tcp [ 1360.218284] LNet: There was an unexpected network error while writing to 192.168.203.158: rc = -22 [ 1360.222171] LNet: Skipped 1 previous similar message [ 1360.498510] LNet: 85535:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1360.503686] LNet: 85535:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1360.546749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.564249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.888338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp -m GET -e remote_dropped -r 2 [ 1360.912620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.158@tcp1 -m GET -e remote_dropped -r 2 [ 1360.941698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp -m GET -e remote_dropped -r 2 [ 1360.980198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.58@tcp -d 192.168.203.58@tcp1 -m GET -e remote_dropped -r 2 [ 1361.004629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp -m GET -e remote_dropped -r 2 [ 1361.030179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.158@tcp1 -m GET -e remote_dropped -r 2 [ 1361.057300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp -m GET -e remote_dropped -r 2 [ 1361.078077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.58@tcp1 -d 192.168.203.58@tcp1 -m GET -e remote_dropped -r 2 [ 1361.107930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.58@tcp 192.168.203.158@tcp [ 1361.123563] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.158@tcp [ 1361.382064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1361.408735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1365.467151] LNet: 85712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.472592] LNetError: 85712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.481423] LNet: Removed LNI 192.168.203.58@tcp [ 1365.494151] LNet: Removed LNI 192.168.203.58@tcp1 [ 1365.867669] Key type .llcrypt unregistered [ 1365.869404] Key type ._llcrypt unregistered [ 1375.403859] Key type ._llcrypt registered [ 1375.405547] Key type .llcrypt registered [ 1376.269445] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:04:56 (1761239096) [ 1376.493989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.505864] alg: No test for adler32 (adler32-zlib) [ 1377.345294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.376558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1377.406808] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1377.925146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1377.969161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1378.000592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1378.041537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1378.068612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1378.101902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1382.166182] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:05:02 (1761239102) [ 1382.327571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.348505] LNet: 87265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.352434] LNetError: 87265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.366785] LNet: Removed LNI 192.168.203.58@tcp [ 1382.381390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.434407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1382.456874] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1386.415349] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:05:06 (1761239106) [ 1386.480880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.500508] LNet: 87961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.505350] LNetError: 87961:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.515322] LNet: Removed LNI 192.168.203.58@tcp [ 1386.538362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.579225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.602779] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1390.528172] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:05:10 (1761239110) [ 1391.529337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1392.576425] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:05:12 (1761239112) [ 1392.893905] LNet: 88996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.899236] LNetError: 88996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.916231] LNet: Removed LNI 192.168.203.58@tcp [ 1393.314592] Key type .llcrypt unregistered [ 1393.316344] Key type ._llcrypt unregistered [ 1396.756346] Key type ._llcrypt registered [ 1396.760672] Key type .llcrypt registered [ 1396.915564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.933500] alg: No test for adler32 (adler32-zlib) [ 1397.793661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.813519] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1402.568951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1406.199299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.158@tcp [ 1406.598981] LNet: 89409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.603514] LNetError: 89409:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.622096] LNet: Removed LNI 192.168.203.58@tcp [ 1407.091498] Key type .llcrypt unregistered [ 1407.093841] Key type ._llcrypt unregistered [ 1414.895388] Key type ._llcrypt registered [ 1414.897872] Key type .llcrypt registered [ 1416.077842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1417.403716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1418.694792] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:05:38 (1761239138) [ 1419.173973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1419.193295] alg: No test for adler32 (adler32-zlib) [ 1420.184483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.293930] LNet: 90494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.889162] Key type .llcrypt unregistered [ 1420.891293] Key type ._llcrypt unregistered [ 1426.190242] Key type ._llcrypt registered [ 1426.192636] Key type .llcrypt registered [ 1426.300599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1426.313935] alg: No test for adler32 (adler32-zlib) [ 1427.185576] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1427.189280] LNet: 90746:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1427.234460] LNetError: 90758:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 1427.246371] LNetError: 90758:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.59@tcp has gone from up to down [ 1427.585403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.687160] LNetError: 90758:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.62@tcp has gone from up to down [ 1430.694356] LNetError: 90758:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 1430.705738] LNet: 90804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.717702] LNetError: 90804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.735148] LNet: Removed LNI 192.168.203.58@tcp [ 1430.762173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.815181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1430.840060] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1430.843894] LNet: 90812:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1430.855500] LNet: 90812:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1430.863044] LNetError: 90812:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.59@tcp is a gateway. Can not delete it [ 1430.873330] LNetError: 90812:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.203.59@tcp is a router. Can not be deleted [ 1430.880355] LNet: 90812:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1430.897182] LNetError: 90809:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 1430.915137] LNetError: 90809:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 1433.382179] LNetError: 90814:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.62@tcp has gone from up to down [ 1433.391800] LNetError: 90814:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1435.566958] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:05:55 (1761239155) [ 1436.825948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg358-server [ 1437.832697] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:05:57 (1761239157) [ 1439.017202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg358-server [ 1440.079552] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:06:00 (1761239160) [ 1441.140891] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1442.928260] Key type lgssc registered [ 1443.734283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.849308] Key type lgssc unregistered [ 1458.088348] LNet: 93227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.096170] LNetError: 93227:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.113083] LNet: Removed LNI 192.168.203.58@tcp [ 1458.519258] Key type .llcrypt unregistered [ 1458.520969] Key type ._llcrypt unregistered [ 1467.589909] Key type ._llcrypt registered [ 1467.591428] Key type .llcrypt registered [ 1468.614911] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:06:28 (1761239188) [ 1469.101621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.116875] alg: No test for adler32 (adler32-zlib) [ 1469.962350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.008322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1470.033956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1470.074500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1470.105442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1474.519181] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:06:34 (1761239194) [ 1474.593100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.609038] LNet: 94627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.624570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.670828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.692678] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1475.237598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.267581] LNet: 94771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.271380] LNetError: 94771:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.282655] LNet: Removed LNI 192.168.203.58@tcp [ 1475.300365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.334314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1475.357651] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1475.660381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.688709] LNet: Removed LNI 192.168.203.58@tcp [ 1475.702897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.737850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1475.757531] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1480.065782] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:06:39 (1761239199) [ 1480.171263] LNet: 95388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.174894] LNet: 95388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1480.183768] LNetError: 95388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.188128] LNetError: 95388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1480.201686] LNet: Removed LNI 192.168.203.58@tcp [ 1480.584842] Key type .llcrypt unregistered [ 1480.586414] Key type ._llcrypt unregistered [ 1483.911570] Key type ._llcrypt registered [ 1483.913832] Key type .llcrypt registered [ 1484.073635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1484.081469] alg: No test for adler32 (adler32-zlib) [ 1484.956641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.004985] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1485.007286] LNet: *** cfs_fail_loc=e003, val=100*** [ 1485.512284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1485.524723] LNet: *** cfs_fail_loc=e003, val=100*** [ 1485.526943] LNet: Skipped 2 previous similar messages [ 1485.534507] Ping reply received with mapping: [ 1485.537315] found metadata reply NI[192.168.203.58@tcp] [ 1485.538974] 3a cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1485.541558] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1485.545743] 5a 5a 5a 5a 5a 5a 5a 5a [ 1485.632148] LNet: 95778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.636526] LNetError: 95778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.649249] LNet: Removed LNI 192.168.203.58@tcp [ 1486.027524] Key type .llcrypt unregistered [ 1486.029260] Key type ._llcrypt unregistered [ 1492.074481] Key type ._llcrypt registered [ 1492.076429] Key type .llcrypt registered [ 1492.754245] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:06:52 (1761239212) [ 1493.051535] Key type .llcrypt unregistered [ 1493.053471] Key type ._llcrypt unregistered [ 1495.979609] Key type ._llcrypt registered [ 1495.981250] Key type .llcrypt registered [ 1496.089683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1496.101221] alg: No test for adler32 (adler32-zlib) [ 1497.092280] LNetError: 96812:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1497.338586] Key type .llcrypt unregistered [ 1497.340492] Key type ._llcrypt unregistered [ 1500.581679] Key type ._llcrypt registered [ 1500.584348] Key type .llcrypt registered [ 1501.454792] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:07:01 (1761239221) [ 1501.737335] Key type .llcrypt unregistered [ 1501.740449] Key type ._llcrypt unregistered [ 1505.292445] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1505.295606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1507.637910] Key type ._llcrypt registered [ 1507.640814] Key type .llcrypt registered [ 1507.773537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.780918] alg: No test for adler32 (adler32-zlib) [ 1508.634175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.679191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1516.227660] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:07:15 (1761239235) [ 1516.363824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.386898] LNet: 98583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.390692] LNetError: 98583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1516.411254] LNet: Removed LNI 10.1.2.3@tcp [ 1516.437995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.930084] Key type .llcrypt unregistered [ 1516.931565] Key type ._llcrypt unregistered [ 1520.301518] Key type ._llcrypt registered [ 1520.303565] Key type .llcrypt registered [ 1520.441933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1520.454574] alg: No test for adler32 (adler32-zlib) [ 1521.306155] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1521.338709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1521.453611] LNet: 98866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.457426] LNetError: 98866:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.467709] LNet: Removed LNI 192.168.203.58@tcp [ 1521.842431] Key type .llcrypt unregistered [ 1521.844083] Key type ._llcrypt unregistered [ 1529.904371] Key type ._llcrypt registered [ 1529.905845] Key type .llcrypt registered [ 1530.855680] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:07:30 (1761239250) [ 1531.377614] Key type .llcrypt unregistered [ 1531.381205] Key type ._llcrypt unregistered [ 1535.380976] Key type ._llcrypt registered [ 1535.382677] Key type .llcrypt registered [ 1535.513456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1535.524882] alg: No test for adler32 (adler32-zlib) [ 1540.175269] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:07:40 (1761239260) [ 1540.344229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.373880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.417831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.448785] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1540.964404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.999399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1541.027728] LNet: Removed LNI 192.168.203.58@tcp [ 1541.234915] LNet: 100598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.520383] Key type .llcrypt unregistered [ 1541.521934] Key type ._llcrypt unregistered [ 1549.014619] Key type ._llcrypt registered [ 1549.017125] Key type .llcrypt registered [ 1549.966433] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:07:49 (1761239269) [ 1550.282261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.296984] alg: No test for adler32 (adler32-zlib) [ 1551.169142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.198355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.217922] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1551.691678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1551.720526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.58@tcp [ 1551.740438] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1551.744241] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1551.746554] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1551, 0, 0 [ 1551.751633] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1551.754680] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1551.759068] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1551.766377] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1551, 0, 0 [ 1551.769051] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1551.772241] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1551.775327] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1551.778032] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1551, 0, 1601 [ 1551.782305] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1551.785246] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1551.788438] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1551.791497] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1551, 0, 1601 [ 1551.797863] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1555.728535] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:07:55 (1761239275) [ 1556.111129] LNet: 102131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.113941] LNetError: 102131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.128031] LNet: Removed LNI 192.168.203.58@tcp [ 1556.581963] Key type .llcrypt unregistered [ 1556.584178] Key type ._llcrypt unregistered [ 1560.126122] Key type ._llcrypt registered [ 1560.135284] Key type .llcrypt registered [ 1560.315609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1560.328781] alg: No test for adler32 (adler32-zlib) [ 1561.189314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1561.209833] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1565.968666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1571.327234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.373140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1571.410052] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1572.115343] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1572.239570] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1572.245200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1572.249558] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1574.431397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1574.454069] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1582.289594] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:08:21 (1761239301) [ 1582.410334] LNet: 103623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.414606] LNetError: 103623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.418315] LNet: Removed LNI 192.168.203.58@tcp [ 1583.457710] LNet: Removed LNI 192.168.203.58@tcp1 [ 1583.459593] LNetError: 103623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1583.463467] LNetError: 103623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1583.478913] LNet: Removed LNI 10.1.2.3@tcp99 [ 1583.887634] Key type .llcrypt unregistered [ 1583.889237] Key type ._llcrypt unregistered [ 1587.693621] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1587.696244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1590.001348] Key type ._llcrypt registered [ 1590.002892] Key type .llcrypt registered [ 1590.112458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1590.122184] alg: No test for adler32 (adler32-zlib) [ 1591.012936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.059256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.082252] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1591.578571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1591.596297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1593.115127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1593.142707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1593.170239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1593.184146] LNetError: 104363:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1593.201653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1593.235844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1593.474076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1593.521785] LNet: 104383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.530919] LNetError: 104383:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.537848] LNet: Removed LNI 192.168.203.58@tcp [ 1593.550759] LNet: Removed LNI 10.1.2.3@tcp [ 1593.961741] Key type .llcrypt unregistered [ 1593.963584] Key type ._llcrypt unregistered [ 1599.561561] Key type ._llcrypt registered [ 1599.563176] Key type .llcrypt registered [ 1600.266642] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:08:40 (1761239320) [ 1600.486108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.495269] alg: No test for adler32 (adler32-zlib) [ 1601.353498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.387779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.410612] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1601.867356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.58@tcp [ 1601.921548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg358-client@tcp [ 1605.680913] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:08:45 (1761239325) [ 1605.864264] LNet: 105917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.868607] LNetError: 105917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.881498] LNet: Removed LNI 192.168.203.58@tcp [ 1606.343648] Key type .llcrypt unregistered [ 1606.345259] Key type ._llcrypt unregistered [ 1610.162874] Key type ._llcrypt registered [ 1610.164764] Key type .llcrypt registered [ 1610.259654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.271635] alg: No test for adler32 (adler32-zlib) [ 1611.115056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1611.132303] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1611.151460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1615.306975] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:08:55 (1761239335) [ 1616.496986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1617.666402] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:08:57 (1761239337) [ 1618.910961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1620.194992] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:08:59 (1761239339) [ 1620.320609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.331623] LNet: 107331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.333926] LNetError: 107331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.361773] LNet: Removed LNI 192.168.203.58@tcp [ 1620.383631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.424618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.439803] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1620.914963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1620.939826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1625.593492] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:09:05 (1761239345) [ 1625.666426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.692590] LNet: 108031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.699287] LNetError: 108031:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.718970] LNet: Removed LNI 192.168.203.58@tcp [ 1625.742707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.776042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1625.801987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1626.458574] Key type .llcrypt unregistered [ 1626.462973] Key type ._llcrypt unregistered [ 1634.996754] Key type ._llcrypt registered [ 1634.998244] Key type .llcrypt registered [ 1636.172170] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:09:15 (1761239355) [ 1636.522879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1636.538904] alg: No test for adler32 (adler32-zlib) [ 1637.396366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.435752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.454768] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1637.978570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1638.062806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1643.781752] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:09:23 (1761239363) [ 1643.904232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.917681] LNet: 109587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.922287] LNetError: 109587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.934125] LNet: Removed LNI 192.168.203.58@tcp [ 1643.946589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.975395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1643.996314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1649.496226] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1650.666965] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:09:30 (1761239370) [ 1650.847625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.864429] LNet: 110308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.881199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.101037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.127397] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1651.732305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1651.748794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1652.389743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.422436] LNet: 110575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.427550] LNetError: 110575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.436523] LNet: Removed LNI 192.168.203.58@tcp [ 1652.464195] LNet: Removed LNI 10.1.2.3@tcp [ 1652.491095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.563850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1652.588969] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1652.597122] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1652.984493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1653.006715] LNet: Removed LNI 192.168.203.58@tcp [ 1653.027278] LNet: Removed LNI 10.1.2.3@tcp [ 1658.461114] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:09:37 (1761239377) [ 1658.828684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.866467] LNet: 111175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.898269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.065361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1659.181273] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1659.183648] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1659.185651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1660.529401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.557426] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1661.253673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1661.296843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1663.800964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.58@tcp [ 1663.852461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1663.907511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.58@tcp -m PUT -e local_timeout -r 1 [ 1663.960440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1664.013679] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1664.016689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1664.292708] LNetError: 111231:0:(lib-move.c:4947:lnet_parse()) 192.168.203.58@tcp, src 192.168.203.58@tcp: Dropping PUT (error -108 looking up sender) [ 1665.055536] LNet: 111683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.068799] LNetError: 111683:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.073168] LNetError: 111683:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1665.101386] LNet: Removed LNI 192.168.203.58@tcp [ 1665.114113] LNet: Removed LNI 10.1.2.3@tcp [ 1665.508242] Key type .llcrypt unregistered [ 1665.509711] Key type ._llcrypt unregistered [ 1673.508713] Key type ._llcrypt registered [ 1673.510266] Key type .llcrypt registered [ 1674.265451] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:09:54 (1761239394) [ 1674.528137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1674.540536] alg: No test for adler32 (adler32-zlib) [ 1675.407217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.690859] LNet: 112531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.210736] Key type .llcrypt unregistered [ 1676.212196] Key type ._llcrypt unregistered [ 1680.526311] Key type ._llcrypt registered [ 1680.527888] Key type .llcrypt registered [ 1680.685714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.695047] alg: No test for adler32 (adler32-zlib) [ 1681.577351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1681.605212] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1686.456303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1686.606800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1687.007734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.158@tcp [ 1687.212760] LNet: 112967:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1687.277362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1687.312151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1687.754088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1687.776081] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.158@tcp [ 1687.951689] LNet: 113072:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1687.996108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1688.020496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1688.424992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.158@tcp [ 1688.727981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1688.755920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1689.195093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1689.417457] LNet: 113282:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1689.420459] LNet: 113282:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1689.470700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1689.491682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1689.725611] LNet: 113323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.730921] LNetError: 113323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.740818] LNet: Removed LNI 192.168.203.58@tcp [ 1690.067690] Key type .llcrypt unregistered [ 1690.069347] Key type ._llcrypt unregistered [ 1698.889264] Key type ._llcrypt registered [ 1698.890904] Key type .llcrypt registered [ 1699.724269] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:10:19 (1761239419) [ 1699.895536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.906967] alg: No test for adler32 (adler32-zlib) [ 1700.758963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.024589] LNet: 114206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.433376] Key type .llcrypt unregistered [ 1701.434974] Key type ._llcrypt unregistered [ 1705.047860] Key type ._llcrypt registered [ 1705.051386] Key type .llcrypt registered [ 1705.246599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1705.280811] alg: No test for adler32 (adler32-zlib) [ 1706.156690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1706.174533] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1710.832355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.158@tcp [ 1715.621820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.643820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1715.662942] LNet: Added LNI 192.168.203.58@tcp1 [8/256/0/180] [ 1716.199454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1716.507966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.158@tcp [ 1716.715693] LNet: 114841:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.769628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.790608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.213287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1717.418953] LNet: 114946:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1717.470792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1717.472257] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.158@tcp, 192.168.203.158@tcp1 [ 1717.505990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.911393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.158@tcp [ 1718.244986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.275950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1718.657681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1718.900395] LNet: 115156:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.904519] LNet: 115156:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1718.945126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.974451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1722.211906] LNet: 115232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.217727] LNetError: 115232:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.222915] LNet: Removed LNI 192.168.203.58@tcp [ 1722.233816] LNet: Removed LNI 192.168.203.58@tcp1 [ 1722.485464] Key type .llcrypt unregistered [ 1722.487140] Key type ._llcrypt unregistered [ 1734.674406] Key type ._llcrypt registered [ 1734.675861] Key type .llcrypt registered [ 1736.168537] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1520 sec ========== 13:10:55 (1761239455) [ 1736.814264] Key type .llcrypt unregistered [ 1736.816382] Key type ._llcrypt unregistered [ 1746.959722] Key type ._llcrypt registered [ 1746.961140] Key type .llcrypt registered [ 1747.160889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.174240] alg: No test for adler32 (adler32-zlib) [ 1748.333128] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1748.511636] LNet: Added LNI 192.168.203.58@tcp [8/256/0/180] [ 1750.223170] Key type lgssc registered [ 1751.511582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1805.166296] Lustre: Mounted lustre-client [ 1809.703207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1820.281964] LustreError: 118454:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff93e7cccb4800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1820.303525] LustreError: 118454:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1820.338587] Lustre: Unmounted lustre-client [ 1839.437480] Key type lgssc unregistered [ 1839.597817] LNet: 118935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.601410] LNetError: 118935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.619526] LNet: Removed LNI 192.168.203.58@tcp [ 1840.159213] Key type .llcrypt unregistered [ 1840.162308] Key type ._llcrypt unregistered