[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506796318 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5410-0x000f541f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5220 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D6F 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C0B 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BCB (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C7F 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D0F 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D47 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c0b-0xbffe1c7e] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c0a] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c7f-0xbffe1d0e] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d0f-0xbffe1d46] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d47-0xbffe1d6e] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007536] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010171] LSM: Security Framework initializing [ 0.011399] Yama: becoming mindful. [ 0.012050] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.022732] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027454] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029036] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031135] *** VALIDATE tmpfs *** [ 0.032631] *** VALIDATE proc *** [ 0.034304] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037046] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045485] debug: unmapping init [mem 0xffffffff8be59000-0xffffffff8be60fff] [ 0.047169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048801] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.050010] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052009] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056508] rcu: Hierarchical SRCU implementation. [ 0.058992] smp: Bringing up secondary CPUs ... [ 0.059731] x86: Booting SMP configuration: [ 0.060021] .... node #0, CPUs: #1 #2 #3 [ 0.186022] smp: Brought up 1 node, 4 CPUs [ 0.188011] smpboot: Max logical packages: 1 [ 0.189018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.261223] node 0 deferred pages initialised in 68ms [ 0.266336] devtmpfs: initialized [ 0.267302] x86/mm: Memory block size: 128MB [ 0.269000] gcov: version magic: 0x41383552 [ 0.272634] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.275111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.278716] pinctrl core: initialized pinctrl subsystem [ 0.281275] [ 0.281781] ************************************************************* [ 0.284020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286013] ** ** [ 0.289016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.291011] ** ** [ 0.293100] ** This means that this kernel is built to expose internal ** [ 0.296013] ** IOMMU data structures, which may compromise security on ** [ 0.298011] ** your system. ** [ 0.301016] ** ** [ 0.304021] ** If you see this message and you are not debugging the ** [ 0.306016] ** kernel, report this immediately to your vendor! ** [ 0.309015] ** ** [ 0.311020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.314012] ************************************************************* [ 0.321517] NET: Registered protocol family 16 [ 0.323714] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.326173] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.329095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.332218] cpuidle: using governor menu [ 0.335057] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.338458] PCI: Using configuration type 1 for base access [ 0.340169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.362044] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.365035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.373675] cryptd: max_cpu_qlen set to 1000 [ 0.384470] ACPI: Added _OSI(Module Device) [ 0.385000] ACPI: Added _OSI(Processor Device) [ 0.387024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.393020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.405988] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.422000] ACPI: Interpreter enabled [ 0.434107] ACPI: PM: (supports S0 S3 S4 S5) [ 0.435022] ACPI: Using IOAPIC for interrupt routing [ 0.436000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.439732] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.456085] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.458103] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.462024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.465093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.471668] acpiphp: Slot [2] registered [ 0.473125] acpiphp: Slot [5] registered [ 0.474204] acpiphp: Slot [6] registered [ 0.480616] acpiphp: Slot [3] registered [ 0.483112] acpiphp: Slot [4] registered [ 0.484194] acpiphp: Slot [7] registered [ 0.485000] acpiphp: Slot [8] registered [ 0.485000] acpiphp: Slot [9] registered [ 0.485000] acpiphp: Slot [10] registered [ 0.486097] acpiphp: Slot [11] registered [ 0.487048] acpiphp: Slot [12] registered [ 0.488000] acpiphp: Slot [13] registered [ 0.489134] acpiphp: Slot [14] registered [ 0.490115] acpiphp: Slot [15] registered [ 0.491122] acpiphp: Slot [16] registered [ 0.492000] acpiphp: Slot [17] registered [ 0.492000] acpiphp: Slot [18] registered [ 0.493133] acpiphp: Slot [19] registered [ 0.495911] acpiphp: Slot [20] registered [ 0.496119] acpiphp: Slot [21] registered [ 0.497128] acpiphp: Slot [22] registered [ 0.498146] acpiphp: Slot [23] registered [ 0.499115] acpiphp: Slot [24] registered [ 0.501112] acpiphp: Slot [25] registered [ 0.503120] acpiphp: Slot [26] registered [ 0.504114] acpiphp: Slot [27] registered [ 0.506138] acpiphp: Slot [28] registered [ 0.507106] acpiphp: Slot [29] registered [ 0.509135] acpiphp: Slot [30] registered [ 0.510097] acpiphp: Slot [31] registered [ 0.512054] PCI host bridge to bus 0000:00 [ 0.513015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.515024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.518020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.520025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.522032] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.525033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.526197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.529200] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.531517] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.537016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.541045] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.544023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.546023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.552024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.553000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.555938] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.559052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.566114] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 10742 usecs [ 0.571834] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.575017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.599755] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.615014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.624246] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.630028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.638021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.659023] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.675467] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.691022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.703032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.734022] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.758055] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.761271] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.764474] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.765403] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.766000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.766182] iommu: Default domain type: Passthrough [ 0.768595] SCSI subsystem initialized [ 0.771422] ACPI: bus type USB registered [ 0.775164] usbcore: registered new interface driver usbfs [ 0.778149] usbcore: registered new interface driver hub [ 0.780132] usbcore: registered new device driver usb [ 0.782224] pps_core: LinuxPPS API ver. 1 registered [ 0.785018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.788087] PTP clock support registered [ 0.789128] EDAC MC: Ver: 3.0.0 [ 0.794140] PCI: Using ACPI for IRQ routing [ 0.798293] NetLabel: Initializing [ 0.799010] NetLabel: domain hash size = 128 [ 0.800000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.800000] NetLabel: unlabeled traffic allowed by default [ 0.800045] vgaarb: loaded [ 0.801000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.804020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.826175] clocksource: Switched to clocksource kvm-clock [ 1.108584] VFS: Disk quotas dquot_6.6.0 [ 1.115617] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.126347] *** VALIDATE ramfs *** [ 1.127691] *** VALIDATE hugetlbfs *** [ 1.131814] pnp: PnP ACPI init [ 1.143711] pnp: PnP ACPI: found 6 devices [ 1.170565] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.177871] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.198356] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.201911] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.208673] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.216757] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.229506] NET: Registered protocol family 2 [ 1.241244] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.264422] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.270168] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.291426] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.308733] TCP: Hash tables configured (established 65536 bind 65536) [ 1.316725] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.325513] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.345986] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.354268] NET: Registered protocol family 1 [ 1.367544] RPC: Registered named UNIX socket transport module. [ 1.373850] RPC: Registered udp transport module. [ 1.378247] RPC: Registered tcp transport module. [ 1.382473] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.387538] NET: Registered protocol family 44 [ 1.391968] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.397298] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.400401] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.403719] PCI: CLS 0 bytes, default 64 [ 1.406546] Unpacking initramfs... [ 5.142467] debug: unmapping init [mem 0xffff996a3cc64000-0xffff996a3ffcffff] [ 5.161110] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.170534] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.180754] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.377365] Initialise system trusted keyrings [ 7.378809] Key type blacklist registered [ 7.382529] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.391797] zbud: loaded [ 7.396401] *** VALIDATE nfs *** [ 7.398180] *** VALIDATE nfs4 *** [ 7.399467] pstore: using deflate compression [ 7.402840] Platform Keyring initialized [ 7.835913] NET: Registered protocol family 38 [ 7.838794] Key type asymmetric registered [ 7.840620] Asymmetric key parser 'x509' registered [ 7.844975] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.851461] io scheduler mq-deadline registered [ 7.853476] io scheduler kyber registered [ 7.855453] io scheduler bfq registered [ 7.857451] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.860915] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.864488] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.868161] ACPI: Power Button [PWRF] [ 8.162090] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.381150] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.881074] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.944358] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.027097] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.037407] Non-volatile memory driver v1.3 [ 9.039498] Linux agpgart interface v0.103 [ 9.121788] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 9.129199] vda: detected capacity change from 0 to 68571136 [ 9.192738] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.196433] vdb: detected capacity change from 0 to 1073741824 [ 9.212646] libphy: Fixed MDIO Bus: probed [ 9.251247] usbcore: registered new interface driver usbserial_generic [ 9.254972] usbserial: USB Serial support registered for generic [ 9.258555] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.270794] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.273215] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.276478] mousedev: PS/2 mouse device common for all mice [ 9.294238] rtc_cmos 00:05: RTC can wake from S4 [ 9.301436] rtc_cmos 00:05: registered as rtc0 [ 9.305383] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.309388] intel_pstate: CPU model not supported [ 9.310253] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.324882] hid: raw HID events driver (C) Jiri Kosina [ 9.331438] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.333197] usbcore: registered new interface driver usbhid [ 9.342710] usbhid: USB HID core driver [ 9.344530] drop_monitor: Initializing network drop monitor service [ 9.344579] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.363953] Initializing XFRM netlink socket [ 9.364381] NET: Registered protocol family 10 [ 9.396613] Segment Routing with IPv6 [ 9.398580] NET: Registered protocol family 17 [ 9.401449] mpls_gso: MPLS GSO support [ 9.406838] RAS: Correctable Errors collector initialized. [ 9.409643] AVX version of gcm_enc/dec engaged. [ 9.412460] AES CTR mode by8 optimization enabled [ 9.713148] sched_clock: Marking stable (9713111208, 0)->(11152868516, -1439757308) [ 9.727785] registered taskstats version 1 [ 9.738277] Loading compiled-in X.509 certificates [ 9.760190] zswap: loaded using pool lzo/zbud [ 9.844030] Key type big_key registered [ 9.868217] Key type encrypted registered [ 9.870623] ima: No TPM chip found, activating TPM-bypass! [ 9.873777] ima: Allocated hash algorithm: sha1 [ 9.875872] ima: No architecture policies found [ 9.878560] evm: Initialising EVM extended attributes: [ 9.881118] evm: security.selinux [ 9.883144] evm: security.ima [ 9.884715] evm: security.capability [ 9.887273] evm: HMAC attrs: 0x1 [ 9.891575] rtc_cmos 00:05: setting system clock to 2025-10-24 05:38:30 UTC (1761284310) [ 9.903881] debug: unmapping init [mem 0xffffffff8ce03000-0xffffffff8cffffff] [ 9.909173] debug: unmapping init [mem 0xffffffff8bb82000-0xffffffff8be58fff] [ 9.919144] Write protecting the kernel read-only data: 28672k [ 9.924579] debug: unmapping init [mem 0xffffffff8a203000-0xffffffff8a3fffff] [ 9.928289] debug: unmapping init [mem 0xffffffff8ab14000-0xffffffff8abfffff] [ 10.076247] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.114198] systemd[1]: Detected virtualization kvm. [ 10.124715] systemd[1]: Detected architecture x86-64. [ 10.128500] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.219175] systemd[1]: No hostname configured. [ 10.226519] systemd[1]: Set hostname to . [ 10.234535] random: systemd: uninitialized urandom read (16 bytes read) [ 10.247861] systemd[1]: Initializing machine ID from random generator. [ 10.748417] random: ln: uninitialized urandom read (6 bytes read) [ 11.430566] random: systemd: uninitialized urandom read (16 bytes read) [ 11.434172] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 11.465634] random: systemd: uninitialized urandom read (16 bytes read) [ 11.471996] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 11.505149] random: systemd: uninitialized urandom read (16 bytes read) [ 11.508693] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.308201] device-mapper: uevent: version 1.0.3 [ 13.310864] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 14.246302] hrtimer: interrupt took 3194166 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 16.942388] virtio_net virtio0 ens2: renamed from eth0 [ 17.007126] random: fast init done [ 18.004207] scsi host0: ata_piix [ 18.119692] scsi host1: ata_piix [ 18.120972] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 18.126922] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 23.471795] random: crng init done [ 23.473462] random: 5 urandom warning(s) missed due to ratelimiting [ 27.452436] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 30.672263] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.307891] printk: systemd: 26 output lines suppressed due to ratelimiting [ 34.039903] SELinux: Disabled at runtime. [ 34.167686] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.188031] systemd[1]: Detected virtualization kvm. [ 34.190429] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 35.826455] systemd[1]: initrd-switch-root.service: Succeeded. [ 35.837411] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 35.857556] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 35.880468] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 35.901756] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 35.930748] systemd[1]: Starting Journal Service... Starting Journal Service... [ 35.957372] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [[ 36.422248] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 37.696878] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.091868] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 39.180591] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 41.711265] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.804581] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 45.213072] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 46.369528] NFS: Registering the id_resolver key type [ 46.378160] Key type id_resolver registered [ 46.384354] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg111-client login: [ 104.710742] libcfs: loading out-of-tree module taints kernel. [ 104.742297] Key type ._llcrypt registered [ 104.744180] Key type .llcrypt registered [ 105.118569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 105.126330] alg: No test for adler32 (adler32-zlib) [ 106.210401] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 106.608284] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 108.256065] Key type lgssc registered [ 109.280182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 177.896528] Lustre: Mounted lustre-client [ 180.762456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 194.649575] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing check_logdir /tmp/testlogs/ [ 197.461938] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing yml_node [ 200.371636] Lustre: DEBUG MARKER: Client: 2.16.59.49 [ 201.968249] Lustre: DEBUG MARKER: MDS: 2.16.59.49 [ 203.350429] Lustre: DEBUG MARKER: OSS: 2.16.59.49 [ 203.746500] Lustre: lustre-OST0000-osc-ffff996a8a439000: disconnect after 24s idle [ 204.228367] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Oct 24 01:41:44 EDT 2025 [ 212.709960] LustreError: 7911:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff996a8a439000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 212.716215] LustreError: 7911:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 212.736210] Lustre: Unmounted lustre-client [ 246.478102] Key type lgssc unregistered [ 246.749253] LNet: 8359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 246.762200] LNetError: 8359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 246.782632] LNet: Removed LNI 192.168.201.11@tcp [ 247.285329] Key type .llcrypt unregistered [ 247.287207] Key type ._llcrypt unregistered [ 262.155565] Key type ._llcrypt registered [ 262.157346] Key type .llcrypt registered [ 262.271673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 262.280528] alg: No test for adler32 (adler32-zlib) [ 263.153805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 263.199731] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 263.214161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 265.616729] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 265.667732] LNet: 9222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 265.674783] LNetError: 9222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 265.685004] LNet: Removed LNI 192.168.201.11@tcp [ 266.049446] Key type .llcrypt unregistered [ 266.052500] Key type ._llcrypt unregistered [ 269.922143] Key type ._llcrypt registered [ 269.923670] Key type .llcrypt registered [ 271.024890] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:42:50 (1761284570) [ 271.407193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 271.434675] alg: No test for adler32 (adler32-zlib) [ 272.342880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.412924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.433742] LNet: 9637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.454464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.497548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 272.515658] LNet: 9644:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 277.282227] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:42:57 (1761284577) [ 277.576175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.608257] LNet: 10221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.638820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.734215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 282.130584] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:43:01 (1761284581) [ 282.250526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.274442] LNet: 10804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.296632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.341873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 286.814391] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:43:06 (1761284586) [ 286.871116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.892278] LNet: 11381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.906043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.970943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 291.371066] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:43:11 (1761284591) [ 291.433889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.463539] LNet: 11966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.481261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.546637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 291.810036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 295.718103] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:43:15 (1761284595) [ 295.811702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.844226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.923291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 296.184894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 296.434550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 300.831265] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:43:20 (1761284600) [ 300.925645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.941909] LNet: 13154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.951441] LNet: 13154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 300.973283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.062813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 306.157364] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:43:25 (1761284605) [ 306.229470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.262401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.304348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 306.328202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 306.586886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 306.613882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 306.831950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 306.852446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 307.057116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 307.083113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 307.290821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 307.314564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 307.530281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 307.547872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 307.784772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 307.810903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 308.094241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 308.121775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 308.384484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 308.405848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 308.688071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 308.711679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 308.939295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 308.962467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 309.229578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 309.257489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 309.518076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 309.544024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 313.954043] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:43:33 (1761284613) [ 314.019826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.059755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.147161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 314.168312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 318.533952] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:43:38 (1761284618) [ 318.650541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.674399] LNet: 15009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.678340] LNet: 15009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 318.704679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.753945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 318.778203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 322.942957] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:43:42 (1761284622) [ 323.017547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.040341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.094722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 323.117593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 327.302128] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:43:46 (1761284626) [ 327.365863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.431452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.514684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 327.555248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 331.773215] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:43:51 (1761284631) [ 331.873648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.897080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.944680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 331.965297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 336.172988] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:43:55 (1761284635) [ 336.258691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.288978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.345246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 336.372103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 340.751234] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:44:00 (1761284640) [ 340.902975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.929625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.999811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 341.277703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 345.297381] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:44:05 (1761284645) [ 345.362878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.388197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.504688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 345.637521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 349.712583] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:44:09 (1761284649) [ 349.850812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.874581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.978803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 350.115456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 354.499646] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:44:14 (1761284654) [ 354.556511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.577090] LNet: 19764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.579842] LNet: 19764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 354.597943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.947985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 355.100491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 359.078061] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:44:18 (1761284658) [ 359.124799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.147610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.189800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.217320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 362.643924] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:44:22 (1761284662) [ 362.706600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.728798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.774938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 362.805737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 366.964830] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:44:26 (1761284666) [ 367.010268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.031706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.089319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 370.962395] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:44:30 (1761284670) [ 371.106833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.132515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.178079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 374.983665] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:44:34 (1761284674) [ 375.066462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.086811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.130238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 375.355702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 378.692433] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:44:38 (1761284678) [ 378.816602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.846652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.894284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 378.913467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 382.517226] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:44:42 (1761284682) [ 382.696554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.730705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.803741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 382.831155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 386.772260] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:44:46 (1761284686) [ 386.822696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.850929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.918116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 386.946273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 390.663766] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:44:50 (1761284690) [ 390.781587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.809347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.869430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 390.889942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 394.871169] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:44:54 (1761284694) [ 395.009082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.037746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.070465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 395.111427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 395.122664] LNetError: 25736:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 395.149965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 395.172971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 398.616494] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:44:58 (1761284698) [ 398.765632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.784600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.849166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.a6UWhfL39X [ 398.908996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.a6UWhfL39X [ 398.938707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.a6UWhfL39X [ 402.464320] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:45:02 (1761284702) [ 402.597377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.625687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.661843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 402.681240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 402.709993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 407.651502] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:45:07 (1761284707) [ 407.889351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.928068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.994833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 409.727551] LNet: Message forwarding will be enabled in 2s [ 412.831112] LNet: Message forwarding enabled [ 413.030251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 413.042782] LNet: Message forwarding will stop in 2s to 6s [ 417.292297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 417.652418] LNet: Message forwarding disabled [ 418.434738] LNet: Message forwarding will be enabled in 2s [ 421.792710] LNet: Message forwarding enabled [ 421.996493] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:45:21 (1761284721) [ 422.037025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.049531] LNet: 28176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.052967] LNet: 28176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 422.698135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.731602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 425.370864] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:45:25 (1761284725) [ 425.410906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.429551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.458253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 425.483022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 425.500441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 425.517667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 425.536136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 425.557798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 425.574568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 425.595113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 425.614455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 425.631595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 425.647452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 425.662756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 425.678763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 425.697295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 425.719378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 425.736325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 425.754542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 425.773922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 425.797485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 425.808256] LNetError: 28804:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 428.593612] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:45:28 (1761284728) [ 428.644734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.664663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.709235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.680249] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:45:32 (1761284732) [ 433.789875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg111-server [ 435.235300] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:45:34 (1761284734) [ 435.570794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.619687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.688950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 435.718604] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 437.163657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[12-16]@tcp [ 437.202262] LNet: 30515:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 437.255749] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.14@tcp is being used as a gateway but routing feature is not turned on [ 437.266428] LNetError: 30308:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.14@tcp has gone from up to down [ 438.256372] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.14@tcp is being used as a gateway but routing feature is not turned on [ 438.272910] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 439.274870] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.14@tcp is being used as a gateway but routing feature is not turned on [ 439.286918] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 440.295854] LNetError: 30312:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.12@tcp has gone from up to down [ 440.340050] LNetError: 30312:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 441.341889] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 441.364783] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 445.485139] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 445.507453] LNetError: 30308:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 445.717697] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:45:44 (1761284744) [ 446.502089] LNetError: 30307:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.201.12@tcp [ 448.079945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg111-server [ 450.914415] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:45:49 (1761284749) [ 451.158228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.679251] LNetError: 30307:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.201.13@tcp [ 454.245860] LNetError: 31451:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.271638] LNet: Removed LNI 192.168.201.11@tcp [ 454.306751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.419163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 454.471502] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 455.146450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[12-17/2]@tcp [ 455.172085] LNet: 31565:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 455.177310] LNet: 31565:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 455.205610] LNetError: 31459:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.12@tcp has gone from up to down [ 455.225610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[12-17/2]@tcp [ 455.230022] LNetError: 31459:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 455.232558] LNetError: 31455:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.14@tcp is being used as a gateway but routing feature is not turned on [ 455.282667] LNetError: 31455:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 17 previous similar messages [ 462.008973] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:46:01 (1761284761) [ 462.472289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.509231] LNetError: 32158:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.528505] LNet: Removed LNI 192.168.201.11@tcp [ 463.569411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.681974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 463.750094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.814536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.878655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.961246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.021059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.071962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 464.193879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.257269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.340680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.470729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.507648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.596454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 464.665907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.728698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.844647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.965608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.031358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.121867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 465.237672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.300390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.401319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.535145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.599740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.669687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 465.778947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.842921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.946914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.058563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.109966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.183305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 466.298748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.408255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.508271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.991821] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:46:12 (1761284772) [ 476.354877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg111-server [ 478.062247] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:46:17 (1761284777) [ 479.762802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg111-server [ 481.510754] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:46:20 (1761284780) [ 482.255749] Key type .llcrypt unregistered [ 482.257364] Key type ._llcrypt unregistered [ 487.066782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 487.127783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 487.137114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 487.141172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 489.546597] Key type ._llcrypt registered [ 489.547908] Key type .llcrypt registered [ 489.690094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 489.702876] alg: No test for adler32 (adler32-zlib) [ 490.589531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.636095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 490.664306] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 491.173440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 491.188508] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 492.654090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 492.668963] LNet: Removed LNI 10.1.2.3@tcp [ 492.751165] LNet: 34269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.760988] LNetError: 34269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.774740] LNet: Removed LNI 192.168.201.11@tcp [ 493.320697] Key type .llcrypt unregistered [ 493.322312] Key type ._llcrypt unregistered [ 500.847939] Key type ._llcrypt registered [ 500.849580] Key type .llcrypt registered [ 501.798989] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:46:41 (1761284801) [ 502.143305] Key type .llcrypt unregistered [ 502.145138] Key type ._llcrypt unregistered [ 505.812987] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 505.903326] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 505.907825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 505.915454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 508.346455] Key type ._llcrypt registered [ 508.347960] Key type .llcrypt registered [ 508.461980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 508.472687] alg: No test for adler32 (adler32-zlib) [ 509.375766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.419117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 509.436222] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 509.967391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 509.981157] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 512.044123] LNet: 35748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.049049] LNetError: 35748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 512.052779] LNet: Removed LNI 192.168.201.11@tcp [ 512.065341] LNet: Removed LNI 10.1.2.3@tcp [ 512.483743] Key type .llcrypt unregistered [ 512.485713] Key type ._llcrypt unregistered [ 520.278639] Key type ._llcrypt registered [ 520.280533] Key type .llcrypt registered [ 521.072380] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:47:00 (1761284820) [ 521.412326] Key type .llcrypt unregistered [ 521.414792] Key type ._llcrypt unregistered [ 525.298635] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 525.369543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 525.373756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 525.379529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 527.753038] Key type ._llcrypt registered [ 527.754516] Key type .llcrypt registered [ 527.877752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 527.886168] alg: No test for adler32 (adler32-zlib) [ 528.741315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.782557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 528.803694] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 529.957958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 529.971686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 531.458682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 531.486235] LNet: Removed LNI 10.1.2.3@tcp [ 531.501143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 531.528730] LNet: Removed LNI 10.1.2.31@tcp [ 531.540704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 531.569814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 532.080325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 532.109355] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 532.807209] LNet: 37648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.811653] LNetError: 37648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 532.815435] LNet: Removed LNI 10.1.2.3@tcp [ 532.826639] LNet: Removed LNI 10.1.2.31@tcp [ 533.288990] Key type .llcrypt unregistered [ 533.290728] Key type ._llcrypt unregistered [ 540.420501] Key type ._llcrypt registered [ 540.421827] Key type .llcrypt registered [ 541.111898] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:47:20 (1761284840) [ 541.351300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 541.361793] alg: No test for adler32 (adler32-zlib) [ 542.203456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.230415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.244537] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 542.722389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.738479] LNet: 38593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.742491] LNetError: 38593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.754464] LNet: Removed LNI 192.168.201.11@tcp [ 542.768335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.810716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.11@tcp [ 542.826485] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 543.253762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.11@tcp [ 543.284640] LNet: Removed LNI 192.168.201.11@tcp [ 543.335395] LNet: 38725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.723449] Key type .llcrypt unregistered [ 543.725142] Key type ._llcrypt unregistered [ 550.033587] Key type ._llcrypt registered [ 550.035432] Key type .llcrypt registered [ 551.101127] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:47:30 (1761284850) [ 552.371628] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg111-server [ 553.635432] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:47:33 (1761284853) [ 554.117583] Key type .llcrypt unregistered [ 554.124613] Key type ._llcrypt unregistered [ 557.964969] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 558.034222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 558.039787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 558.045910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 560.540719] Key type ._llcrypt registered [ 560.545970] Key type .llcrypt registered [ 560.658821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 560.669790] alg: No test for adler32 (adler32-zlib) [ 561.522411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.569834] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 562.056148] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 563.497634] LNet: 40546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.502038] LNetError: 40546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.505533] LNet: Removed LNI 192.168.201.11@tcp1 [ 563.517083] LNet: Removed LNI 10.1.2.3@tcp2 [ 563.919456] Key type .llcrypt unregistered [ 563.921644] Key type ._llcrypt unregistered [ 571.694945] Key type ._llcrypt registered [ 571.696291] Key type .llcrypt registered [ 572.690384] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:47:52 (1761284872) [ 573.034535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 573.050950] alg: No test for adler32 (adler32-zlib) [ 573.947782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.001667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 574.018946] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 574.637823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.656128] LNet: 41508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.660809] LNetError: 41508:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.675576] LNet: Removed LNI 192.168.201.11@tcp [ 574.694992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.763172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 574.787866] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 578.807694] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:47:58 (1761284878) [ 578.934086] LNet: 42097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.937995] LNetError: 42097:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.946712] LNet: Removed LNI 192.168.201.11@tcp [ 579.353687] Key type .llcrypt unregistered [ 579.355333] Key type ._llcrypt unregistered [ 582.976450] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 583.105149] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 583.107663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 583.141724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 585.492111] Key type ._llcrypt registered [ 585.493729] Key type .llcrypt registered [ 585.651829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 585.664516] alg: No test for adler32 (adler32-zlib) [ 586.573418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.613464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.638463] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 587.201408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 587.228828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.822243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.834145] LNet: 42831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.839793] LNetError: 42831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.843142] LNet: Removed LNI 192.168.201.11@tcp [ 588.848588] LNet: Removed LNI 10.1.2.3@tcp [ 588.861117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.898848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 588.927080] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 588.932740] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 589.262182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.276873] LNet: Removed LNI 192.168.201.11@tcp [ 589.290832] LNet: Removed LNI 10.1.2.3@tcp [ 589.304412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.365436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 589.395073] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 589.399203] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 589.631704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 589.652299] LNet: Added LNI 192.168.201.11@tcp2 [8/256/0/180] [ 590.289752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.302578] LNet: 43017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.305571] LNet: 43017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 590.308256] LNetError: 43017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.312359] LNetError: 43017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 590.317151] LNet: Removed LNI 192.168.201.11@tcp [ 590.318958] LNet: Removed LNI 10.1.2.3@tcp [ 590.326466] LNet: Removed LNI 192.168.201.11@tcp2 [ 590.340753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.383351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 590.397904] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 590.402243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 590.404907] LNet: Added LNI 192.168.201.11@tcp2 [8/256/0/180] [ 595.084586] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:48:14 (1761284894) [ 595.249937] LNet: 43615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.259810] LNetError: 43615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.272848] LNetError: 43615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 595.280573] LNet: Removed LNI 192.168.201.11@tcp [ 595.283334] LNet: Removed LNI 10.1.2.3@tcp [ 595.297183] LNet: Removed LNI 192.168.201.11@tcp2 [ 595.735607] Key type .llcrypt unregistered [ 595.737129] Key type ._llcrypt unregistered [ 599.766914] Key type ._llcrypt registered [ 599.768369] Key type .llcrypt registered [ 599.910694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 599.926876] alg: No test for adler32 (adler32-zlib) [ 600.801851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.852165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 600.884465] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 601.510385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.525651] LNet: 44002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.531820] LNetError: 44002:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.545117] LNet: Removed LNI 192.168.201.11@tcp [ 601.566417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.616083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 601.649581] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 601.974515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.997841] LNet: Removed LNI 192.168.201.11@tcp [ 602.010580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.052736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 602.075641] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 602.432989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.451568] LNet: 44062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.455360] LNet: 44062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 602.459719] LNetError: 44062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.466688] LNetError: 44062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 602.499417] LNet: Removed LNI 192.168.201.11@tcp [ 602.532279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.619683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 602.649173] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 602.978911] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 603.018567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 603.021448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 603.023540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 605.481506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.498164] LNet: 44141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.505588] LNetError: 44141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.526569] LNet: Removed LNI 192.168.201.11@tcp [ 605.550426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.605366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 605.651491] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 610.689644] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:48:30 (1761284910) [ 610.871488] LNet: 44737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.877709] LNetError: 44737:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.894754] LNet: Removed LNI 192.168.201.11@tcp [ 611.479232] Key type .llcrypt unregistered [ 611.484075] Key type ._llcrypt unregistered [ 615.636530] Key type ._llcrypt registered [ 615.640324] Key type .llcrypt registered [ 615.770232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 615.780248] alg: No test for adler32 (adler32-zlib) [ 616.639421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.719102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 616.741637] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 617.468567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.489842] LNet: 45128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.501439] LNetError: 45128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.510627] LNet: Removed LNI 192.168.201.11@tcp [ 617.542172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.600151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 617.628210] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 623.138567] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:48:42 (1761284922) [ 623.239478] LNet: 45711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.251720] LNetError: 45711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.295873] LNet: Removed LNI 192.168.201.11@tcp [ 623.894726] Key type .llcrypt unregistered [ 623.896518] Key type ._llcrypt unregistered [ 628.249623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 628.329538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 628.334324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 628.337910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 630.888338] Key type ._llcrypt registered [ 630.889970] Key type .llcrypt registered [ 631.093755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 631.114104] alg: No test for adler32 (adler32-zlib) [ 632.135518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.186634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.212628] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 633.054721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.076126] LNet: 46161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.081326] LNetError: 46161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.097230] LNet: Removed LNI 192.168.201.11@tcp [ 633.114942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.207555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 633.245542] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 638.629612] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:48:58 (1761284938) [ 638.817175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.835240] LNet: 46759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.840207] LNetError: 46759:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.856558] LNet: Removed LNI 192.168.201.11@tcp [ 638.873505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.963990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 643.179676] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:49:02 (1761284942) [ 643.353948] LNet: 47341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.811622] Key type .llcrypt unregistered [ 643.813314] Key type ._llcrypt unregistered [ 648.156035] Key type ._llcrypt registered [ 648.157390] Key type .llcrypt registered [ 648.303050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 648.315594] alg: No test for adler32 (adler32-zlib) [ 649.192128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.231989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.256281] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 649.872888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.884184] LNet: 47731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.887322] LNetError: 47731:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.902791] LNet: Removed LNI 192.168.201.11@tcp [ 649.919137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.973679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 649.996386] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 654.584160] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:49:14 (1761284954) [ 654.725215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.751214] LNet: 48323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.755845] LNetError: 48323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.769926] LNet: Removed LNI 192.168.201.11@tcp [ 654.790237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.861907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 659.348765] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:49:19 (1761284959) [ 659.382359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.393037] LNet: 48893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.410177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.445351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.472191] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 659.891838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.11 [ 659.908619] LNet: Removed LNI 192.168.201.11@tcp [ 663.853311] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:49:23 (1761284963) [ 663.951109] LNet: 49597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.955966] LNetError: 49597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.974537] LNet: Removed LNI 192.168.201.11@tcp [ 664.472633] Key type .llcrypt unregistered [ 664.476838] Key type ._llcrypt unregistered [ 669.011719] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 669.249829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 669.253139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 669.263781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 671.799141] Key type ._llcrypt registered [ 671.800679] Key type .llcrypt registered [ 671.904902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.925242] alg: No test for adler32 (adler32-zlib) [ 673.070321] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 677.679811] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:49:37 (1761284977) [ 677.811276] LNet: 50499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.833693] LNetError: 50499:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 677.861104] LNet: Removed LNI 10.1.2.3@tcp [ 678.416734] Key type .llcrypt unregistered [ 678.418498] Key type ._llcrypt unregistered [ 682.901960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 683.092477] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 683.095139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 683.098210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 685.632879] Key type ._llcrypt registered [ 685.634829] Key type .llcrypt registered [ 685.771413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.783769] alg: No test for adler32 (adler32-zlib) [ 686.704171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.412549] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:49:50 (1761284990) [ 690.574279] LNet: 51395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.580925] LNetError: 51395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 690.598909] LNet: Removed LNI 10.1.2.3@tcp [ 691.069673] Key type .llcrypt unregistered [ 691.071934] Key type ._llcrypt unregistered [ 694.715419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.894988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.902116] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.905385] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 697.366178] Key type ._llcrypt registered [ 697.369075] Key type .llcrypt registered [ 697.514422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.533537] alg: No test for adler32 (adler32-zlib) [ 698.489374] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.099473] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:50:02 (1761285002) [ 703.271692] LNet: 52283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.277707] LNetError: 52283:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 703.293968] LNet: Removed LNI 10.1.2.3@tcp [ 703.761780] Key type .llcrypt unregistered [ 703.767447] Key type ._llcrypt unregistered [ 707.592175] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 707.751105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 707.756936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 707.760599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 710.274531] Key type ._llcrypt registered [ 710.275887] Key type .llcrypt registered [ 710.392520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 710.404824] alg: No test for adler32 (adler32-zlib) [ 711.266340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.318865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.213091] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:50:16 (1761285016) [ 717.642553] LNet: 53504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.647197] LNetError: 53504:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 717.661094] LNet: Removed LNI 10.1.2.3@tcp [ 718.079382] Key type .llcrypt unregistered [ 718.080906] Key type ._llcrypt unregistered [ 721.350901] Key type ._llcrypt registered [ 721.352413] Key type .llcrypt registered [ 721.467326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 721.487240] alg: No test for adler32 (adler32-zlib) [ 722.339770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 722.355430] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 726.834601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 727.320839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_interrupt -r 1 [ 727.349618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_interrupt -r 1 [ 727.370748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 727.384958] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 727.575782] LNet: 53928:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 727.621912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 727.645946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 727.673111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 727.969165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_dropped -r 1 [ 727.989402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_dropped -r 1 [ 728.012583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 728.025250] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 728.222214] LNet: 54025:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 728.263938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 728.285844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 728.302666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 728.560334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_aborted -r 1 [ 728.583184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_aborted -r 1 [ 728.603503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 728.824957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 728.844294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 728.864655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 729.132749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_no_route -r 1 [ 729.161863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_no_route -r 1 [ 729.185245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 729.200351] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 729.203667] LNet: Skipped 1 previous similar message [ 729.362631] LNet: 54220:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 729.366613] LNet: 54220:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 729.412711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 729.430609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 729.452148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 729.741080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_timeout -r 1 [ 729.764832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_timeout -r 1 [ 729.793366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 730.056857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 730.084524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 730.102632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 730.494040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_error -r 1 [ 730.520304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_error -r 1 [ 730.548627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 730.814697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 730.838338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 730.859504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 731.035518] LNet: 54450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.043212] LNetError: 54450:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.060690] LNet: Removed LNI 192.168.201.11@tcp [ 731.293281] Key type .llcrypt unregistered [ 731.297648] Key type ._llcrypt unregistered [ 740.889399] Key type ._llcrypt registered [ 740.890801] Key type .llcrypt registered [ 741.887581] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:50:41 (1761285041) [ 742.499314] Key type .llcrypt unregistered [ 742.504802] Key type ._llcrypt unregistered [ 746.479776] Key type ._llcrypt registered [ 746.483143] Key type .llcrypt registered [ 746.616907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 746.624625] alg: No test for adler32 (adler32-zlib) [ 747.469572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 747.491353] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 752.728722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 761.338906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.379025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 761.405143] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 762.628413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_interrupt -r 1 [ 762.658367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_interrupt -r 1 [ 762.706825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_interrupt -r 1 [ 762.751840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_interrupt -r 1 [ 762.831171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_interrupt -r 1 [ 762.885667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_interrupt -r 1 [ 762.945412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_interrupt -r 1 [ 763.013463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_interrupt -r 1 [ 763.071607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 763.095134] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 763.099769] LNet: 1 local NIs in recovery (showing 1): 192.168.201.11@tcp [ 763.411317] LNet: 55897:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 763.507756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 763.530337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 763.552582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 763.995255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_dropped -r 1 [ 764.033398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_dropped -r 1 [ 764.058454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_dropped -r 1 [ 764.078972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_dropped -r 1 [ 764.124722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_dropped -r 1 [ 764.162810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_dropped -r 1 [ 764.212655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_dropped -r 1 [ 764.246560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_dropped -r 1 [ 764.277360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 764.295234] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 764.300527] LNet: Skipped 2 previous similar messages [ 764.584364] LNet: 56006:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 764.645574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 764.663354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 764.700541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 765.209782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_aborted -r 1 [ 765.239541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_aborted -r 1 [ 765.273265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_aborted -r 1 [ 765.304696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_aborted -r 1 [ 765.336594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_aborted -r 1 [ 765.364288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_aborted -r 1 [ 765.419307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_aborted -r 1 [ 765.461139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_aborted -r 1 [ 765.534781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 765.550524] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 765.554325] LNet: Skipped 2 previous similar messages [ 765.807147] LNet: 56115:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 765.864415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 765.893057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 765.924850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 766.341257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_no_route -r 1 [ 766.377201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_no_route -r 1 [ 766.397759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_no_route -r 1 [ 766.431278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_no_route -r 1 [ 766.456189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_no_route -r 1 [ 766.487130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_no_route -r 1 [ 766.526140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_no_route -r 1 [ 766.560242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_no_route -r 1 [ 766.589285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 766.948066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 766.969775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 767.002578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 767.482698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_timeout -r 1 [ 767.522942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_timeout -r 1 [ 767.563120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_timeout -r 1 [ 767.620638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_timeout -r 1 [ 767.670594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_timeout -r 1 [ 767.723973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_timeout -r 1 [ 767.767578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_timeout -r 1 [ 767.808451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_timeout -r 1 [ 767.834515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 767.850584] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 767.856622] LNet: Skipped 6 previous similar messages [ 768.105231] LNet: 56333:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 768.108730] LNet: 56333:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 768.161620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 768.192377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 768.215692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 768.686363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_error -r 1 [ 768.717658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_error -r 1 [ 768.752604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_error -r 1 [ 768.782827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_error -r 1 [ 768.832776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_error -r 1 [ 768.855695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_error -r 1 [ 768.891459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_error -r 1 [ 768.935675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_error -r 1 [ 768.971639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 769.316189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 769.336147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 769.359621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 773.081466] LNet: 56513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.089393] LNetError: 56513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.099874] LNet: Removed LNI 192.168.201.11@tcp [ 774.126464] LNet: Removed LNI 192.168.201.11@tcp1 [ 774.605090] Key type .llcrypt unregistered [ 774.606820] Key type ._llcrypt unregistered [ 786.903936] Key type ._llcrypt registered [ 786.905515] Key type .llcrypt registered [ 787.866578] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:51:27 (1761285087) [ 788.588525] Key type .llcrypt unregistered [ 788.590264] Key type ._llcrypt unregistered [ 792.076480] Key type ._llcrypt registered [ 792.078101] Key type .llcrypt registered [ 792.281378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 792.310539] alg: No test for adler32 (adler32-zlib) [ 793.219066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 793.283154] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 799.980289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 800.538413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_dropped -r 1 [ 800.570104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_dropped -r 1 [ 800.620890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 800.638748] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 800.917706] LNet: 57746:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 801.028602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 801.100115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 801.139482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 801.688771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_error -r 1 [ 801.755970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_error -r 1 [ 801.802568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 801.819280] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 802.101584] LNet: 57844:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 802.189620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 802.217487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 802.249400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 802.704809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_timeout -r 1 [ 802.745959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_timeout -r 1 [ 802.789602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 803.122803] LNet: 57941:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 803.177974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 803.205626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 803.228400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 803.420571] LNet: 57977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.424486] LNetError: 57977:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.461682] LNet: Removed LNI 192.168.201.11@tcp [ 804.816187] Key type .llcrypt unregistered [ 804.817984] Key type ._llcrypt unregistered [ 817.650766] Key type ._llcrypt registered [ 817.653035] Key type .llcrypt registered [ 818.691741] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:51:58 (1761285118) [ 819.366581] Key type .llcrypt unregistered [ 819.368146] Key type ._llcrypt unregistered [ 823.530072] Key type ._llcrypt registered [ 823.531444] Key type .llcrypt registered [ 823.697829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.710530] alg: No test for adler32 (adler32-zlib) [ 824.645101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 824.673851] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 830.475413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 836.711351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.757855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 836.780593] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 837.973456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_dropped -r 1 [ 838.000516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e remote_dropped -r 1 [ 838.023675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_dropped -r 1 [ 838.054882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e remote_dropped -r 1 [ 838.078894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e remote_dropped -r 1 [ 838.108841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e remote_dropped -r 1 [ 838.130866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e remote_dropped -r 1 [ 838.166869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e remote_dropped -r 1 [ 838.208106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 838.221251] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 838.228621] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.111@tcp [ 838.471879] LNet: 59425:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 838.541631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.565341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.593334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 838.635842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.134743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_error -r 1 [ 839.170191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e remote_error -r 1 [ 839.202913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_error -r 1 [ 839.239423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e remote_error -r 1 [ 839.275165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e remote_error -r 1 [ 839.309220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e remote_error -r 1 [ 839.342477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e remote_error -r 1 [ 839.370429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e remote_error -r 1 [ 839.407397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 839.423871] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 839.428265] LNet: Skipped 2 previous similar messages [ 839.661630] LNet: 59536:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.712616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.748664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.782361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 839.803699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 840.195468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_timeout -r 1 [ 840.238503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e remote_timeout -r 1 [ 840.286526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_timeout -r 1 [ 840.332186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e remote_timeout -r 1 [ 840.359832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e remote_timeout -r 1 [ 840.389703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e remote_timeout -r 1 [ 840.416777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e remote_timeout -r 1 [ 840.441398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e remote_timeout -r 1 [ 840.471429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.111@tcp [ 840.493830] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 840.717116] LNet: 59647:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 840.767384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 840.797411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 840.829394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 840.860899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 844.106924] LNet: 59720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.113389] LNetError: 59720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.123031] LNet: Removed LNI 192.168.201.11@tcp [ 844.139128] LNet: Removed LNI 192.168.201.11@tcp1 [ 844.393496] Key type .llcrypt unregistered [ 844.398240] Key type ._llcrypt unregistered [ 853.934198] Key type ._llcrypt registered [ 853.935802] Key type .llcrypt registered [ 854.832286] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:52:34 (1761285154) [ 855.216677] Key type .llcrypt unregistered [ 855.219382] Key type ._llcrypt unregistered [ 858.928519] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 859.016454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 859.018786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 859.021083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 860.409178] Key type ._llcrypt registered [ 860.410860] Key type .llcrypt registered [ 860.558144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.571673] alg: No test for adler32 (adler32-zlib) [ 861.472144] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 861.496400] LNet: 60790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.501460] LNetError: 60790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.511807] LNet: Removed LNI 192.168.201.11@tcp [ 862.078436] Key type .llcrypt unregistered [ 862.084273] Key type ._llcrypt unregistered [ 866.060619] Key type ._llcrypt registered [ 866.062487] Key type .llcrypt registered [ 866.175704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.197260] alg: No test for adler32 (adler32-zlib) [ 867.045367] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 867.047804] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 867.073890] LNet: 61059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.079308] LNetError: 61059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.083501] LNet: Removed LNI 192.168.201.11@tcp [ 867.095799] LNet: Removed LNI 10.1.2.3@tcp1 [ 867.549701] Key type .llcrypt unregistered [ 867.551144] Key type ._llcrypt unregistered [ 871.408858] Key type ._llcrypt registered [ 871.410360] Key type .llcrypt registered [ 871.535984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.543751] alg: No test for adler32 (adler32-zlib) [ 872.446268] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 872.449314] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.475972] LNet: 61328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.480347] LNetError: 61328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.488830] LNet: Removed LNI 192.168.201.11@tcp [ 872.502279] LNet: Removed LNI 10.1.2.3@tcp [ 872.933373] Key type .llcrypt unregistered [ 872.936964] Key type ._llcrypt unregistered [ 876.491876] Key type ._llcrypt registered [ 876.493571] Key type .llcrypt registered [ 876.625755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.637749] alg: No test for adler32 (adler32-zlib) [ 877.480674] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 877.483044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 877.503303] LNet: 61598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.507573] LNetError: 61598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.512289] LNet: Removed LNI 192.168.201.11@tcp [ 877.520654] LNet: Removed LNI 10.1.2.3@tcp [ 877.883375] Key type .llcrypt unregistered [ 877.885654] Key type ._llcrypt unregistered [ 880.947701] Key type ._llcrypt registered [ 880.949852] Key type .llcrypt registered [ 881.127241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.139345] alg: No test for adler32 (adler32-zlib) [ 882.000117] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 882.002199] LNetError: 61856:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 882.006148] LNetError: Error -100 starting up LNI tcp [ 882.014748] LNet: Removed LNI 192.168.201.11@tcp [ 882.416233] Key type .llcrypt unregistered [ 882.418727] Key type ._llcrypt unregistered [ 888.926288] Key type ._llcrypt registered [ 888.927921] Key type .llcrypt registered [ 889.900756] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:53:09 (1761285189) [ 890.509328] Key type .llcrypt unregistered [ 890.511052] Key type ._llcrypt unregistered [ 894.288172] Key type ._llcrypt registered [ 894.289923] Key type .llcrypt registered [ 894.511959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.524564] alg: No test for adler32 (adler32-zlib) [ 895.401856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 895.421369] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 900.306784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 900.659444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e network_timeout -r 1 [ 900.687252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e network_timeout -r 1 [ 900.717173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 900.734250] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 900.972206] LNet: 63096:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 901.014037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.036282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.196086] LNet: 63130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.199701] LNetError: 63130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.211453] LNet: Removed LNI 192.168.201.11@tcp [ 901.435759] Key type .llcrypt unregistered [ 901.437512] Key type ._llcrypt unregistered [ 911.830852] Key type ._llcrypt registered [ 911.832472] Key type .llcrypt registered [ 911.965860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.989201] alg: No test for adler32 (adler32-zlib) [ 912.862837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 912.886710] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 918.087591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 924.164416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.228770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 924.278510] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 925.462543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e network_timeout -r 1 [ 925.491827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e network_timeout -r 1 [ 925.518723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e network_timeout -r 1 [ 925.573098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e network_timeout -r 1 [ 925.627248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e network_timeout -r 1 [ 925.666440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e network_timeout -r 1 [ 925.716704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e network_timeout -r 1 [ 925.752342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e network_timeout -r 1 [ 925.783599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 925.809282] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 926.158938] LNet: 63991:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.239599] LNet: 1 local NIs in recovery (showing 1): 192.168.201.11@tcp [ 926.251654] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.111@tcp [ 926.254864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.317677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.098064] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 930.102171] LNet: Skipped 2 previous similar messages [ 930.319909] LNet: 64060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.324157] LNetError: 64060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.329403] LNet: Removed LNI 192.168.201.11@tcp [ 930.347866] LNet: Removed LNI 192.168.201.11@tcp1 [ 930.686686] Key type .llcrypt unregistered [ 930.689625] Key type ._llcrypt unregistered [ 942.846366] Key type ._llcrypt registered [ 942.847738] Key type .llcrypt registered [ 943.924911] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:54:03 (1761285243) [ 944.182329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.198327] alg: No test for adler32 (adler32-zlib) [ 945.064910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.106198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.133520] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 945.632738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 945.654055] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 946.229261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 946.277272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 946.307068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 946.332533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 946.363751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 946.390405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 946.415246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.11@tcp [ 946.434364] LNetError: 65149:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.201.11@tcp to recovery [ 947.168780] LNet: 1 local NIs in recovery (showing 1): 192.168.201.11@tcp [ 947.169189] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 949.215454] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 949.759079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 953.311328] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 954.098377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 954.120704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 954.153023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.171252] LNet: 65267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.174763] LNetError: 65267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.178049] LNet: Removed LNI 192.168.201.11@tcp [ 954.187321] LNet: Removed LNI 192.168.201.11@tcp1 [ 954.199550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.232212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 954.246551] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 954.661912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 954.672407] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 955.181322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 955.201434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.269299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 955.298636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 955.318948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 955.349247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 955.373865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 955.397576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.11@tcp [ 955.409257] LNetError: 65533:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.201.11@tcp to recovery [ 956.255402] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 958.719793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 962.399454] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 962.406015] LNet: Skipped 1 previous similar message [ 967.372682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 967.393360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 967.413839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.780497] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:54:31 (1761285271) [ 971.879350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.901834] LNet: 66255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.909920] LNetError: 66255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.913599] LNetError: 66255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 972.959265] LNet: Removed LNI 192.168.201.11@tcp [ 972.961174] LNetError: 66255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.997941] LNet: Removed LNI 192.168.201.11@tcp1 [ 973.023976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.068901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.100326] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 973.564726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 973.576113] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 974.242798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 974.295741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 974.334288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 974.355859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 974.382583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 974.420532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 974.448645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.11@tcp --health 0 [ 974.510378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 975.071311] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.11@tcp [ 975.071687] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 975.084115] LNet: Skipped 1 previous similar message [ 990.520728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 990.658794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 990.692533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.734439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.11@tcp --health 500 [ 990.811162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 997.339990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 997.391752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.404795] LNet: 66753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.411119] LNetError: 66753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.415298] LNet: Removed LNI 192.168.201.11@tcp [ 997.424303] LNet: Removed LNI 192.168.201.11@tcp1 [ 997.446056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.494411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.513668] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 998.045726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 998.068534] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 998.646085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 998.680292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 998.743310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 998.766784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 998.788977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 998.816829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 998.842497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 998.872409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.11@tcp --health 0 [ 999.519436] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 999.526862] LNet: Skipped 3 previous similar messages [ 1000.089253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1010.919776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1010.946836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1010.975679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.840234] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:55:15 (1761285315) [ 1016.313477] LNet: 67805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.322489] LNetError: 67805:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.325904] LNetError: 67805:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1016.332754] LNet: Removed LNI 192.168.201.11@tcp [ 1016.349839] LNet: Removed LNI 192.168.201.11@tcp1 [ 1016.847259] Key type .llcrypt unregistered [ 1016.848482] Key type ._llcrypt unregistered [ 1021.739865] Key type ._llcrypt registered [ 1021.741619] Key type .llcrypt registered [ 1021.975630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.999361] alg: No test for adler32 (adler32-zlib) [ 1022.958261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1022.990868] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1030.318692] Lustre: DEBUG MARKER: Initial discovery [ 1030.514932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.111@tcp [ 1032.544567] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1032.703083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1032.752959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.111@tcp [ 1032.767627] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 1032.785768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1032.808155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.111@tcp -r 1 -m GET -l 3 [ 1032.855987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.111@tcp -r 1 -m GET -e local_error [ 1032.900273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.111@tcp -r 1 -m PUT -l 6 [ 1033.999618] Lustre: DEBUG MARKER: Force oleg111-client.virtnet to discover 192.168.201.111@tcp (in background) [ 1034.172180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.111@tcp [ 1035.210393] Lustre: DEBUG MARKER: Force oleg111-server to discover 192.168.201.11@tcp [ 1037.216486] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 1037.229176] Lustre: DEBUG MARKER: Wait for 68563 [ 1038.447505] Lustre: DEBUG MARKER: Finished wait on 68563 [ 1038.614451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1038.648930] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 1038.659730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1038.888690] LNet: 69011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.898205] LNetError: 69011:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.916599] LNet: Removed LNI 192.168.201.11@tcp [ 1039.232551] Key type .llcrypt unregistered [ 1039.234218] Key type ._llcrypt unregistered [ 1052.668808] Key type ._llcrypt registered [ 1052.670212] Key type .llcrypt registered [ 1053.897808] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:55:53 (1761285353) [ 1054.346312] Key type .llcrypt unregistered [ 1054.347984] Key type ._llcrypt unregistered [ 1058.606209] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1058.646387] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1058.649241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1058.651966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1061.125742] Key type ._llcrypt registered [ 1061.127300] Key type .llcrypt registered [ 1061.357764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.370915] alg: No test for adler32 (adler32-zlib) [ 1062.231724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.278754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.303524] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1062.808859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1062.830369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1070.147575] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:56:09 (1761285369) [ 1070.486082] LNet: 71075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.507690] LNetError: 71075:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.535907] LNet: Removed LNI 192.168.201.11@tcp [ 1070.569109] LNet: Removed LNI 10.1.2.3@tcp [ 1071.223868] Key type .llcrypt unregistered [ 1071.225704] Key type ._llcrypt unregistered [ 1076.435592] Key type ._llcrypt registered [ 1076.439878] Key type .llcrypt registered [ 1076.656849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1076.678841] alg: No test for adler32 (adler32-zlib) [ 1077.562691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.626987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1077.670813] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1078.471373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1078.491660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1079.633634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.11@tcp [ 1080.737805] LNetError: Unexpected error -22 connecting to 192.168.201.11@tcp at host 192.168.201.11:988 [ 1086.386719] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:56:25 (1761285385) [ 1086.669036] LNet: 72196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.685478] LNetError: 72196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.692135] LNet: Removed LNI 192.168.201.11@tcp [ 1086.703487] LNet: Removed LNI 10.1.2.3@tcp [ 1087.357996] Key type .llcrypt unregistered [ 1087.359863] Key type ._llcrypt unregistered [ 1091.639514] Key type ._llcrypt registered [ 1091.641330] Key type .llcrypt registered [ 1091.783804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.799962] alg: No test for adler32 (adler32-zlib) [ 1092.714961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.769701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1092.788114] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 1093.521518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1093.544628] LNet: Added LNI 192.168.201.11@tcp2 [8/256/0/180] [ 1094.319791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.11@tcp1 --nid 192.168.201.11@tcp2 [ 1098.142213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp1 192.168.201.11@tcp2 [ 1098.159497] LNetError: 73021:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.11@tcp1 to 192.168.201.11@tcp2 - Route table may be misconfigured [ 1098.177441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp2 192.168.201.11@tcp1 [ 1103.238738] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:56:42 (1761285402) [ 1103.389681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1103.411655] LNet: 73594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.424932] LNetError: 73594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.428877] LNet: Removed LNI 192.168.201.11@tcp1 [ 1104.488353] LNet: Removed LNI 192.168.201.11@tcp2 [ 1104.515735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.559700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1104.587577] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1105.180488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1105.200493] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 1105.817967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 1105.845761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.11@tcp [ 1105.872680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp1 192.168.201.11@tcp1 [ 1105.897352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.11@tcp -d 192.168.201.11@tcp -e network_timeout [ 1105.917812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -e network_timeout [ 1105.947570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -e network_timeout [ 1105.981195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -e network_timeout [ 1106.015195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1106.030468] LNet: There was an unexpected network error while writing to 192.168.201.11: rc = -22 [ 1106.046839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1106.134904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1106.210741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1106.591125] LNet: 1 local NIs in recovery (showing 1): 192.168.201.11@tcp [ 1111.538744] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:56:51 (1761285411) [ 1111.696277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1111.731110] LNet: 74449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.744470] LNetError: 74449:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.753389] LNetError: 74449:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1111.775732] LNet: Removed LNI 192.168.201.11@tcp [ 1112.808771] LNet: Removed LNI 192.168.201.11@tcp1 [ 1112.859843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.992851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1113.288239] LNet: 74486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.650780] Key type .llcrypt unregistered [ 1113.656741] Key type ._llcrypt unregistered [ 1122.256295] Key type ._llcrypt registered [ 1122.257873] Key type .llcrypt registered [ 1123.221684] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:57:02 (1761285422) [ 1123.533032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1123.548486] alg: No test for adler32 (adler32-zlib) [ 1124.433446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.613051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1127.163732] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:57:06 (1761285426) [ 1127.332464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.345566] LNet: 75581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.359896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.408035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1127.424949] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1127.929933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.943036] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 1128.603762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1128.634922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp1 [ 1128.667317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp1 [ 1133.466383] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:57:13 (1761285433) [ 1135.018577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg111-server [ 1136.265727] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:57:15 (1761285435) [ 1137.858671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg111-server [ 1139.312984] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:57:18 (1761285438) [ 1141.008678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg111-server [ 1142.443637] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:57:21 (1761285441) [ 1143.983603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg111-server [ 1145.298723] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:57:24 (1761285444) [ 1147.097727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg111-server [ 1148.550636] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:57:28 (1761285448) [ 1149.997410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg111-server [ 1151.199761] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:57:30 (1761285450) [ 1152.619265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg111-server [ 1153.824310] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:57:33 (1761285453) [ 1155.236349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg111-server [ 1156.545381] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:57:36 (1761285456) [ 1158.004917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg111-server [ 1159.345274] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:57:38 (1761285458) [ 1159.675918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1159.707879] LNet: 79450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.712267] LNetError: 79450:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.720134] LNet: Removed LNI 192.168.201.11@tcp [ 1160.744454] LNet: Removed LNI 192.168.201.11@tcp1 [ 1160.768720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1160.820733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1160.844178] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1161.371205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1161.428905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1161.550394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1161.571295] LNet: 79594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.575902] LNetError: 79594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.582702] LNetError: 79594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1161.598785] LNet: Removed LNI 192.168.201.11@tcp [ 1161.611806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.652528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1161.678304] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1162.269344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1163.508471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.531582] LNetError: 79746:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.544169] LNet: Removed LNI 192.168.201.11@tcp [ 1163.568032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.613259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1163.639908] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1164.133094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1164.211600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1164.362346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1164.376320] LNet: 79890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.379967] LNet: 79890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1165.413845] LNet: Removed LNI 192.168.201.11@tcp [ 1165.442179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.485826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1165.520972] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1166.119604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1166.320616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1166.359239] LNetError: 80032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.367387] LNetError: 80032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1166.389656] LNet: Removed LNI 192.168.201.11@tcp [ 1166.420352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1166.497737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1166.525026] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1167.098089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1167.182688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1167.352812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1168.427986] LNet: Removed LNI 192.168.201.11@tcp [ 1168.453678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.512257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1168.558971] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1169.215789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1169.411941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1169.427476] LNet: 80319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.431748] LNet: 80319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1170.471265] LNet: Removed LNI 192.168.201.11@tcp [ 1170.494271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.544526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1170.566963] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1171.075417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1171.153931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1171.329988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.343212] LNetError: 80463:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.352972] LNetError: 80463:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1171.365706] LNet: Removed LNI 192.168.201.11@tcp [ 1171.380354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.434614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1171.456552] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1172.089559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1173.362224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1174.439169] LNet: Removed LNI 192.168.201.11@tcp [ 1174.456551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1174.510171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1174.541994] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1175.055239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1175.115663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1175.300216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1176.360560] LNet: Removed LNI 192.168.201.11@tcp [ 1176.377187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.415786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1176.434237] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1176.969512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1177.147929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1178.216165] LNet: Removed LNI 192.168.201.11@tcp [ 1178.243489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.295942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1178.332523] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1178.928937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1179.002751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1179.222370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1179.238849] LNet: 81045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.242820] LNet: 81045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1180.263210] LNet: Removed LNI 192.168.201.11@tcp [ 1180.299887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.342810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1180.380597] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1180.954405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1181.154610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1181.173544] LNetError: 81187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.178779] LNetError: 81187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1181.189608] LNet: Removed LNI 192.168.201.11@tcp [ 1181.207462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.251036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1181.269996] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1181.812591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1181.844404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1181.875449] LNet: Removed LNI 192.168.201.11@tcp [ 1181.897152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.959505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1181.976297] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1182.589069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1187.477590] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:58:07 (1761285487) [ 1187.699811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1187.722695] LNet: Removed LNI 192.168.201.11@tcp [ 1187.743146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.785184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1187.807708] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/180] [ 1188.572795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1188.599290] LNet: Removed LNI 192.168.201.11@tcp231 [ 1188.615712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.670124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1188.692317] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/180] [ 1188.698472] LNet: 82160:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.015128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1189.034539] LNet: Removed LNI 192.168.201.11@tcp231 [ 1189.055400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1189.079138] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/180] [ 1189.804644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1189.833411] LNet: Removed LNI 192.168.201.11@tcp231 [ 1189.849627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.915797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1189.945380] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/0] [ 1189.948933] LNet: 82310:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.215564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1190.238675] LNet: Removed LNI 192.168.201.11@tcp231 [ 1190.253187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1190.274800] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/0] [ 1191.139110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1191.174587] LNet: Removed LNI 192.168.201.11@tcp231 [ 1191.194535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.241274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1191.262334] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/60] [ 1191.272844] LNet: 82458:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.554263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1191.581769] LNet: Removed LNI 192.168.201.11@tcp231 [ 1191.597824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1191.622735] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/60] [ 1192.328565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1192.348215] LNet: Removed LNI 192.168.201.11@tcp231 [ 1192.364195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.414447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1192.431260] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/180] [ 1192.719417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1192.744185] LNet: Removed LNI 192.168.201.11@tcp231 [ 1192.761701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1192.785719] LNet: Added LNI 192.168.201.11@tcp231 [8/256/0/180] [ 1197.583778] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:58:17 (1761285497) [ 1197.770096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1197.781726] LNet: 83314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.784832] LNet: 83314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1197.792063] LNetError: 83314:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.795117] LNetError: 83314:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1197.812849] LNet: Removed LNI 192.168.201.11@tcp231 [ 1197.828141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1197.867448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1197.882850] LNet: Added LNI 192.168.201.11@tcp232 [8/256/0/180] [ 1198.314535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1198.347226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1198.366543] LNet: Removed LNI 192.168.201.11@tcp232 [ 1198.378181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1198.408391] LNet: Added LNI 192.168.201.11@tcp232 [8/256/0/180] [ 1198.829406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1198.851665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1198.871117] LNet: Removed LNI 192.168.201.11@tcp232 [ 1198.884508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1198.917286] LNet: Added LNI 192.168.201.11@tcp232 [8/256/0/180] [ 1199.378631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1199.398503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1199.424152] LNet: Removed LNI 192.168.201.11@tcp232 [ 1199.433208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1199.453402] LNet: Added LNI 192.168.201.11@tcp232 [8/256/0/180] [ 1199.881155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1199.914367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1199.934801] LNet: Removed LNI 192.168.201.11@tcp232 [ 1199.947910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1203.575448] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:58:23 (1761285503) [ 1204.248738] Key type .llcrypt unregistered [ 1204.251506] Key type ._llcrypt unregistered [ 1207.730366] Key type ._llcrypt registered [ 1207.731991] Key type .llcrypt registered [ 1207.956776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.980591] alg: No test for adler32 (adler32-zlib) [ 1208.846188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1208.864616] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1213.528790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 1217.492367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.515413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1217.529600] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 1218.226564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_interrupt -r 2 [ 1218.245912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_interrupt -r 2 [ 1218.267815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_interrupt -r 2 [ 1218.290505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_interrupt -r 2 [ 1218.313927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_interrupt -r 2 [ 1218.337359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_interrupt -r 2 [ 1218.360819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_interrupt -r 2 [ 1218.380121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_interrupt -r 2 [ 1218.401732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.111@tcp [ 1218.412794] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 1218.415118] LNet: 1 local NIs in recovery (showing 1): 192.168.201.11@tcp [ 1218.573181] LNet: 85076:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.610346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.635385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.902142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_dropped -r 2 [ 1218.929843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_dropped -r 2 [ 1218.954280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_dropped -r 2 [ 1218.975836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_dropped -r 2 [ 1219.001718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_dropped -r 2 [ 1219.025870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_dropped -r 2 [ 1219.045699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_dropped -r 2 [ 1219.068037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_dropped -r 2 [ 1219.087591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.111@tcp [ 1219.103199] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 1219.297958] LNet: 85184:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.340611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.369252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.651264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_aborted -r 2 [ 1219.670437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_aborted -r 2 [ 1219.693913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_aborted -r 2 [ 1219.718665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_aborted -r 2 [ 1219.742756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_aborted -r 2 [ 1219.767534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_aborted -r 2 [ 1219.786721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_aborted -r 2 [ 1219.814248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_aborted -r 2 [ 1219.838703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.111@tcp [ 1220.085784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.113513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1220.391240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_no_route -r 2 [ 1220.417433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_no_route -r 2 [ 1220.439791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_no_route -r 2 [ 1220.465754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_no_route -r 2 [ 1220.488763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_no_route -r 2 [ 1220.512677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_no_route -r 2 [ 1220.534473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_no_route -r 2 [ 1220.556330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_no_route -r 2 [ 1220.576608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.111@tcp [ 1220.592506] LNet: There was an unexpected network error while writing to 192.168.201.111: rc = -22 [ 1220.596912] LNet: Skipped 2 previous similar messages [ 1220.759406] LNet: 85401:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.761724] LNet: 85401:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1220.801325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.819435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.103341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e local_timeout -r 2 [ 1221.121775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e local_timeout -r 2 [ 1221.141988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e local_timeout -r 2 [ 1221.163133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e local_timeout -r 2 [ 1221.181839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e local_timeout -r 2 [ 1221.206587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e local_timeout -r 2 [ 1221.228657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e local_timeout -r 2 [ 1221.253844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e local_timeout -r 2 [ 1221.280984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.111@tcp [ 1221.515474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.541840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.830315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp -m GET -e remote_dropped -r 2 [ 1221.849707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp -d 192.168.201.111@tcp1 -m GET -e remote_dropped -r 2 [ 1221.874658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp -m GET -e remote_dropped -r 2 [ 1221.899307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp -d 192.168.201.11@tcp1 -m GET -e remote_dropped -r 2 [ 1221.927212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp -m GET -e remote_dropped -r 2 [ 1221.949747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.111@tcp1 -m GET -e remote_dropped -r 2 [ 1221.973411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp -m GET -e remote_dropped -r 2 [ 1221.994679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.11@tcp1 -d 192.168.201.11@tcp1 -m GET -e remote_dropped -r 2 [ 1222.032702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.11@tcp 192.168.201.111@tcp [ 1222.052861] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.111@tcp [ 1222.268514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.293757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.232339] LNet: 85686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.237876] LNetError: 85686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.271163] LNet: Removed LNI 192.168.201.11@tcp [ 1226.273844] LNetError: 85686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.284152] LNet: Removed LNI 192.168.201.11@tcp1 [ 1226.499220] Key type .llcrypt unregistered [ 1226.500864] Key type ._llcrypt unregistered [ 1236.406390] Key type ._llcrypt registered [ 1236.407883] Key type .llcrypt registered [ 1237.392192] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:58:56 (1761285536) [ 1237.865897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1237.877761] alg: No test for adler32 (adler32-zlib) [ 1238.761602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.809743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1238.842727] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1239.420958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1239.488055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1239.544991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1239.604605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1239.649325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1239.678315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1243.990717] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:59:03 (1761285543) [ 1244.170570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1244.187438] LNet: 87239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.193488] LNetError: 87239:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.207752] LNet: Removed LNI 192.168.201.11@tcp [ 1244.219847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.262204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1244.287420] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1249.153313] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:59:08 (1761285548) [ 1249.216972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1249.234984] LNet: 87932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.241205] LNetError: 87932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.254940] LNet: Removed LNI 192.168.201.11@tcp [ 1249.280290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.337515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1249.355644] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1254.162728] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:59:13 (1761285553) [ 1255.041417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1255.984361] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:59:15 (1761285555) [ 1256.211954] LNet: 88971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.215441] LNetError: 88971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.224498] LNet: Removed LNI 192.168.201.11@tcp [ 1256.522112] Key type .llcrypt unregistered [ 1256.523605] Key type ._llcrypt unregistered [ 1259.733426] Key type ._llcrypt registered [ 1259.735029] Key type .llcrypt registered [ 1259.870597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1259.884145] alg: No test for adler32 (adler32-zlib) [ 1260.728395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1260.747963] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1265.490924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 1268.678213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.111@tcp [ 1268.948590] LNet: 89381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.956568] LNetError: 89381:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.964711] LNet: Removed LNI 192.168.201.11@tcp [ 1269.231396] Key type .llcrypt unregistered [ 1269.233142] Key type ._llcrypt unregistered [ 1275.269508] Key type ._llcrypt registered [ 1275.272079] Key type .llcrypt registered [ 1275.934713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1276.801217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1277.795817] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:59:37 (1761285577) [ 1278.011816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.020913] alg: No test for adler32 (adler32-zlib) [ 1278.873758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.937081] LNet: 90464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.312096] Key type .llcrypt unregistered [ 1279.314675] Key type ._llcrypt unregistered [ 1282.313840] Key type ._llcrypt registered [ 1282.315385] Key type .llcrypt registered [ 1282.439310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.453687] alg: No test for adler32 (adler32-zlib) [ 1283.293701] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1283.296828] LNet: 90715:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1283.314098] LNetError: 90727:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.15@tcp is being used as a gateway but routing feature is not turned on [ 1283.329932] LNetError: 90727:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.201.15@tcp has gone from up to down [ 1283.644707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1283.656208] LNet: 90773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.660056] LNetError: 90773:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.674172] LNet: Removed LNI 192.168.201.11@tcp [ 1283.691420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1283.723600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1283.739358] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1283.748935] LNetError: 90780:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.13@tcp is a gateway. Can not delete it [ 1283.757220] LNetError: 90780:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.201.13@tcp is a router. Can not be deleted [ 1283.765245] LNet: 90780:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.872289] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:59:47 (1761285587) [ 1288.964709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg111-server [ 1290.254832] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:59:49 (1761285589) [ 1291.417674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg111-server [ 1292.555344] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:59:52 (1761285592) [ 1293.617056] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 1295.415166] Key type lgssc registered [ 1296.158511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1308.371252] Key type lgssc unregistered [ 1308.631322] LNet: 93200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.639101] LNetError: 93200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.664161] LNet: Removed LNI 192.168.201.11@tcp [ 1309.008963] Key type .llcrypt unregistered [ 1309.010707] Key type ._llcrypt unregistered [ 1317.918989] Key type ._llcrypt registered [ 1317.920991] Key type .llcrypt registered [ 1318.708800] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:00:18 (1761285618) [ 1318.940501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.948730] alg: No test for adler32 (adler32-zlib) [ 1319.807329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.847946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1319.875786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1319.916589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1319.945638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1324.998595] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:00:24 (1761285624) [ 1325.203941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.218753] LNet: 94603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.235604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.301630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.322048] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1326.017437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.037934] LNet: 94744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.042494] LNetError: 94744:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.053834] LNet: Removed LNI 192.168.201.11@tcp [ 1326.075363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.154984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1326.171600] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1326.501796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.529573] LNet: Removed LNI 192.168.201.11@tcp [ 1326.550434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.592512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1326.618916] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1332.062726] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:00:31 (1761285631) [ 1332.124884] LNet: 95353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.127992] LNet: 95353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1332.136982] LNetError: 95353:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.146264] LNetError: 95353:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1332.155917] LNet: Removed LNI 192.168.201.11@tcp [ 1332.614745] Key type .llcrypt unregistered [ 1332.617467] Key type ._llcrypt unregistered [ 1337.006735] Key type ._llcrypt registered [ 1337.008707] Key type .llcrypt registered [ 1337.275356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1337.295039] alg: No test for adler32 (adler32-zlib) [ 1338.177891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.242967] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1338.245164] LNet: *** cfs_fail_loc=e003, val=100*** [ 1338.945743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1338.962808] LNet: *** cfs_fail_loc=e003, val=100*** [ 1338.964849] LNet: Skipped 2 previous similar messages [ 1338.971414] Ping reply received with mapping: [ 1338.973489] found metadata reply NI[192.168.201.11@tcp] [ 1338.976792] 0b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1338.982658] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1338.984418] 5a 5a 5a 5a 5a 5a 5a 5a [ 1339.067416] LNet: 95752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.071854] LNetError: 95752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.133863] LNet: Removed LNI 192.168.201.11@tcp [ 1340.522484] Key type .llcrypt unregistered [ 1340.524611] Key type ._llcrypt unregistered [ 1346.662757] Key type ._llcrypt registered [ 1346.664498] Key type .llcrypt registered [ 1347.622162] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:00:47 (1761285647) [ 1348.024128] Key type .llcrypt unregistered [ 1348.025851] Key type ._llcrypt unregistered [ 1351.819650] Key type ._llcrypt registered [ 1351.820656] Key type .llcrypt registered [ 1351.926870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1351.961215] alg: No test for adler32 (adler32-zlib) [ 1352.895649] LNetError: 96786:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1353.142945] Key type .llcrypt unregistered [ 1353.144542] Key type ._llcrypt unregistered [ 1356.196816] Key type ._llcrypt registered [ 1356.198422] Key type .llcrypt registered [ 1357.001090] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:00:56 (1761285656) [ 1357.381176] Key type .llcrypt unregistered [ 1357.382690] Key type ._llcrypt unregistered [ 1361.480621] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1361.490627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1364.080100] Key type ._llcrypt registered [ 1364.081507] Key type .llcrypt registered [ 1364.286986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1364.307195] alg: No test for adler32 (adler32-zlib) [ 1365.228343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.306777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1373.448980] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:01:12 (1761285672) [ 1373.522536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.543947] LNet: 98563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.549230] LNetError: 98563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1373.573368] LNet: Removed LNI 10.1.2.3@tcp [ 1373.603211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.114369] Key type .llcrypt unregistered [ 1374.116539] Key type ._llcrypt unregistered [ 1378.285057] Key type ._llcrypt registered [ 1378.290267] Key type .llcrypt registered [ 1378.453037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1378.464866] alg: No test for adler32 (adler32-zlib) [ 1379.410780] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1379.475885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1379.630428] LNet: 98855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.644103] LNetError: 98855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.711844] LNet: Removed LNI 192.168.201.11@tcp [ 1381.361536] Key type .llcrypt unregistered [ 1381.363317] Key type ._llcrypt unregistered [ 1389.207256] Key type ._llcrypt registered [ 1389.209197] Key type .llcrypt registered [ 1390.107451] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:01:29 (1761285689) [ 1390.623596] Key type .llcrypt unregistered [ 1390.625396] Key type ._llcrypt unregistered [ 1394.990585] Key type ._llcrypt registered [ 1394.992276] Key type .llcrypt registered [ 1395.197173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1395.210234] alg: No test for adler32 (adler32-zlib) [ 1400.602986] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:01:40 (1761285700) [ 1400.693115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.715984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.782777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1400.807884] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1401.337555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.392106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1401.418606] LNet: Removed LNI 192.168.201.11@tcp [ 1401.565842] LNet: 100586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.853408] Key type .llcrypt unregistered [ 1401.854996] Key type ._llcrypt unregistered [ 1410.519543] Key type ._llcrypt registered [ 1410.521163] Key type .llcrypt registered [ 1411.645967] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:01:51 (1761285711) [ 1411.983924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1411.999847] alg: No test for adler32 (adler32-zlib) [ 1412.940338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.995412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.019750] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1413.658621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1413.691836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.11@tcp [ 1413.717490] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1413.720219] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1413.723538] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1413, 0, 0 [ 1413.728728] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1413.731000] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1413.737707] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1413.741804] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1413, 0, 0 [ 1413.747330] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1413.749928] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1413.761671] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1413.768241] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1413, 0, 1463 [ 1413.771426] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1413.777249] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1413.781560] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1413.789328] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1413, 0, 1463 [ 1413.799188] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1418.848807] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:01:58 (1761285718) [ 1419.267591] LNet: 102121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.277369] LNetError: 102121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.289135] LNet: Removed LNI 192.168.201.11@tcp [ 1419.823671] Key type .llcrypt unregistered [ 1419.825518] Key type ._llcrypt unregistered [ 1424.541428] Key type ._llcrypt registered [ 1424.551124] Key type .llcrypt registered [ 1424.769461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1424.784666] alg: No test for adler32 (adler32-zlib) [ 1425.704350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1425.740382] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1431.889348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 1438.485800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.532885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1438.549806] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 1439.383619] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1439.528773] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1439.532679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1439.536720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1441.814824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1441.832110] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1450.445268] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:02:30 (1761285750) [ 1450.645615] LNet: 103614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.651052] LNetError: 103614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.658103] LNet: Removed LNI 192.168.201.11@tcp [ 1451.679218] LNet: Removed LNI 192.168.201.11@tcp1 [ 1451.683059] LNetError: 103614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1451.698949] LNetError: 103614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1451.716913] LNet: Removed LNI 10.1.2.3@tcp99 [ 1452.189967] Key type .llcrypt unregistered [ 1452.192883] Key type ._llcrypt unregistered [ 1456.747764] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1456.750403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1459.171685] Key type ._llcrypt registered [ 1459.174964] Key type .llcrypt registered [ 1459.300482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.314899] alg: No test for adler32 (adler32-zlib) [ 1460.283596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.393131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.417718] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1461.049866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1461.072210] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1462.562421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1462.602086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1462.649400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1462.665977] LNetError: 104350:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1462.697584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1462.730966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 1463.015822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1463.055220] LNet: 104370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.061746] LNetError: 104370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.095182] LNet: Removed LNI 192.168.201.11@tcp [ 1464.100751] LNetError: 104370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1464.112067] LNet: Removed LNI 10.1.2.3@tcp [ 1464.593495] Key type .llcrypt unregistered [ 1464.595659] Key type ._llcrypt unregistered [ 1475.277695] Key type ._llcrypt registered [ 1475.279268] Key type .llcrypt registered [ 1476.553800] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:02:55 (1761285775) [ 1477.124122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1477.169133] alg: No test for adler32 (adler32-zlib) [ 1478.117480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.182195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1478.216413] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1479.047684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.11@tcp [ 1479.119928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg111-client@tcp [ 1484.587502] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:03:04 (1761285784) [ 1484.653674] LNet: 105889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.664675] LNetError: 105889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.675797] LNet: Removed LNI 192.168.201.11@tcp [ 1485.241815] Key type .llcrypt unregistered [ 1485.243711] Key type ._llcrypt unregistered [ 1489.957464] Key type ._llcrypt registered [ 1489.959015] Key type .llcrypt registered [ 1490.133064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1490.154973] alg: No test for adler32 (adler32-zlib) [ 1491.059938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1491.087862] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1491.122722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1496.149661] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:03:15 (1761285795) [ 1497.482843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1499.213497] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:03:18 (1761285798) [ 1500.695941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1502.256296] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:03:21 (1761285801) [ 1502.608877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.634786] LNet: 107312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.644408] LNetError: 107312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.671701] LNet: Removed LNI 192.168.201.11@tcp [ 1502.699050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.780799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.822453] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1503.745552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1503.808539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1509.580236] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:03:29 (1761285809) [ 1509.698120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.724679] LNet: 108014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.731895] LNetError: 108014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.744922] LNet: Removed LNI 192.168.201.11@tcp [ 1509.762230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.833193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1509.856415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1510.545566] Key type .llcrypt unregistered [ 1510.549334] Key type ._llcrypt unregistered [ 1518.618779] Key type ._llcrypt registered [ 1518.620691] Key type .llcrypt registered [ 1519.484222] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:03:39 (1761285819) [ 1519.944471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.954680] alg: No test for adler32 (adler32-zlib) [ 1520.867544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.928589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1520.949588] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1521.462837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1521.519162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 1525.676153] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:03:45 (1761285825) [ 1525.734601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.748341] LNet: 109557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.753456] LNetError: 109557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.766959] LNet: Removed LNI 192.168.201.11@tcp [ 1525.799167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.846931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1525.867226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1530.303170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1531.451915] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:03:51 (1761285831) [ 1531.589225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.615439] LNet: 110284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.657265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.921709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1531.943136] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1532.571467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1532.588455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1533.412248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.432787] LNet: 110552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.436334] LNetError: 110552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.439382] LNet: Removed LNI 192.168.201.11@tcp [ 1533.452642] LNet: Removed LNI 10.1.2.3@tcp [ 1533.480590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.565754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1533.604232] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1533.610135] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1534.058613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1534.072921] LNet: Removed LNI 192.168.201.11@tcp [ 1534.083843] LNet: Removed LNI 10.1.2.3@tcp [ 1539.886318] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:03:59 (1761285839) [ 1540.034488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.049474] LNet: 111150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.070748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.140161] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1540.235667] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1540.239700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1540.242387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1542.563557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.591726] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1543.080547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1543.094552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1544.468610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.11@tcp [ 1544.499779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1544.544462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.11@tcp -m PUT -e local_timeout -r 1 [ 1544.595358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1544.644732] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1544.650087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1544.812280] LNetError: 111218:0:(lib-move.c:4947:lnet_parse()) 192.168.201.11@tcp, src 192.168.201.11@tcp: Dropping PUT (error -108 looking up sender) [ 1545.828626] LNet: 111672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.835535] LNetError: 111672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.841060] LNetError: 111672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1545.844653] LNet: Removed LNI 192.168.201.11@tcp [ 1545.862686] LNet: Removed LNI 10.1.2.3@tcp [ 1546.200641] Key type .llcrypt unregistered [ 1546.206202] Key type ._llcrypt unregistered [ 1555.002610] Key type ._llcrypt registered [ 1555.003971] Key type .llcrypt registered [ 1556.080107] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:04:15 (1761285855) [ 1556.406248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.427063] alg: No test for adler32 (adler32-zlib) [ 1557.300253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.639381] LNet: 112521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.144352] Key type .llcrypt unregistered [ 1558.146646] Key type ._llcrypt unregistered [ 1563.636571] Key type ._llcrypt registered [ 1563.647730] Key type .llcrypt registered [ 1563.865077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.888575] alg: No test for adler32 (adler32-zlib) [ 1564.829797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1564.850724] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1571.697992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 1571.931767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1572.549750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.111@tcp [ 1572.901548] LNet: 112959:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.953597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1572.983585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1573.587623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1573.997526] LNet: 113065:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1574.047726] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.111@tcp [ 1574.076405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1574.105992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1574.692181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.111@tcp [ 1575.053654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1575.100516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1575.771381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1576.046392] LNet: 113275:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1576.050166] LNet: 113275:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1576.102851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1576.124329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1576.310715] LNet: 113316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.315901] LNetError: 113316:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.328953] LNet: Removed LNI 192.168.201.11@tcp [ 1576.611893] Key type .llcrypt unregistered [ 1576.620048] Key type ._llcrypt unregistered [ 1585.272861] Key type ._llcrypt registered [ 1585.274209] Key type .llcrypt registered [ 1586.095587] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:04:45 (1761285885) [ 1586.421282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.432330] alg: No test for adler32 (adler32-zlib) [ 1587.299031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.603461] LNet: 114199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.059452] Key type .llcrypt unregistered [ 1588.061239] Key type ._llcrypt unregistered [ 1592.208842] Key type ._llcrypt registered [ 1592.210448] Key type .llcrypt registered [ 1592.444799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1592.463489] alg: No test for adler32 (adler32-zlib) [ 1593.439640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1593.461226] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1599.601572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.111@tcp [ 1605.623987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.683390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1605.716319] LNet: Added LNI 192.168.201.11@tcp1 [8/256/0/180] [ 1606.477100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1606.925331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.111@tcp [ 1607.228716] LNet: 114836:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1607.278858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1607.312626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1607.803850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1607.840332] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.111@tcp, 192.168.201.111@tcp1 [ 1608.027129] LNet: 114941:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1608.094225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1608.125198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1608.623442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.111@tcp [ 1608.939982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1608.973624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1609.338846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1609.516984] LNet: 115152:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1609.524135] LNet: 115152:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1609.576298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1609.604128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1612.689476] LNet: 115227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.697903] LNetError: 115227:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.710936] LNet: Removed LNI 192.168.201.11@tcp [ 1612.720616] LNet: Removed LNI 192.168.201.11@tcp1 [ 1613.029503] Key type .llcrypt unregistered [ 1613.031536] Key type ._llcrypt unregistered [ 1623.612177] Key type ._llcrypt registered [ 1623.613891] Key type .llcrypt registered [ 1624.536754] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1420 sec ========== 02:05:24 (1761285924) [ 1625.121579] Key type .llcrypt unregistered [ 1625.140331] Key type ._llcrypt unregistered [ 1635.257526] Key type ._llcrypt registered [ 1635.258733] Key type .llcrypt registered [ 1635.709816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1635.727315] alg: No test for adler32 (adler32-zlib) [ 1636.772685] Lustre: Lustre: Build Version: 2.16.59_49_g4265256 [ 1637.013189] LNet: Added LNI 192.168.201.11@tcp [8/256/0/180] [ 1638.679164] Key type lgssc registered [ 1639.729865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1686.460824] Lustre: Mounted lustre-client [ 1690.477561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1701.049837] LustreError: 118448:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff996a898c5000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1701.070032] LustreError: 118448:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1701.101041] Lustre: Unmounted lustre-client [ 1720.595456] Key type lgssc unregistered [ 1720.796621] LNet: 118929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.804987] LNetError: 118929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.827724] LNet: Removed LNI 192.168.201.11@tcp [ 1721.306156] Key type .llcrypt unregistered [ 1721.307862] Key type ._llcrypt unregistered