[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500841133 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002425] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007454] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.012070] LSM: Security Framework initializing [ 0.013079] Yama: becoming mindful. [ 0.014049] SELinux: Initializing. [ 0.015071] *** VALIDATE selinux *** [ 0.024205] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028819] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031135] *** VALIDATE tmpfs *** [ 0.033003] *** VALIDATE proc *** [ 0.034120] *** VALIDATE cgroup *** [ 0.035021] *** VALIDATE cgroup2 *** [ 0.036320] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038027] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039042] Spectre V2 : User space: Vulnerable [ 0.040022] Speculative Store Bypass: Vulnerable [ 0.043682] debug: unmapping init [mem 0xffffffff89e59000-0xffffffff89e60fff] [ 0.046210] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048114] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049041] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052015] ... value mask: 0000ffffffffffff [ 0.053032] ... max period: 00007fffffffffff [ 0.054022] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056392] rcu: Hierarchical SRCU implementation. [ 0.058568] smp: Bringing up secondary CPUs ... [ 0.059779] x86: Booting SMP configuration: [ 0.060040] .... node #0, CPUs: #1 #2 #3 [ 0.065164] smp: Brought up 1 node, 4 CPUs [ 0.067017] smpboot: Max logical packages: 1 [ 0.068026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.285510] node 0 deferred pages initialised in 216ms [ 0.289014] devtmpfs: initialized [ 0.290291] x86/mm: Memory block size: 128MB [ 0.292678] gcov: version magic: 0x41383552 [ 0.293687] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.296131] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.298437] pinctrl core: initialized pinctrl subsystem [ 0.300208] [ 0.300671] ************************************************************* [ 0.303020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304014] ** ** [ 0.306017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.308016] ** ** [ 0.310015] ** This means that this kernel is built to expose internal ** [ 0.312016] ** IOMMU data structures, which may compromise security on ** [ 0.314022] ** your system. ** [ 0.315015] ** ** [ 0.317017] ** If you see this message and you are not debugging the ** [ 0.319029] ** kernel, report this immediately to your vendor! ** [ 0.321019] ** ** [ 0.322016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.324018] ************************************************************* [ 0.326833] NET: Registered protocol family 16 [ 0.328532] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.330075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.333072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.336017] cpuidle: using governor menu [ 0.339649] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.342144] PCI: Using configuration type 1 for base access [ 0.344139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.353186] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.356068] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.360162] cryptd: max_cpu_qlen set to 1000 [ 0.362278] ACPI: Added _OSI(Module Device) [ 0.363019] ACPI: Added _OSI(Processor Device) [ 0.364011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.365014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.369595] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.376650] ACPI: Interpreter enabled [ 0.378072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.379010] ACPI: Using IOAPIC for interrupt routing [ 0.380103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.382405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.392033] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.394042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.395022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.398088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.402209] acpiphp: Slot [2] registered [ 0.403134] acpiphp: Slot [3] registered [ 0.404128] acpiphp: Slot [4] registered [ 0.405081] acpiphp: Slot [5] registered [ 0.406143] acpiphp: Slot [6] registered [ 0.407118] acpiphp: Slot [7] registered [ 0.408077] acpiphp: Slot [8] registered [ 0.409105] acpiphp: Slot [9] registered [ 0.410077] acpiphp: Slot [10] registered [ 0.411103] acpiphp: Slot [11] registered [ 0.413083] acpiphp: Slot [12] registered [ 0.414094] acpiphp: Slot [13] registered [ 0.415098] acpiphp: Slot [14] registered [ 0.416082] acpiphp: Slot [15] registered [ 0.417082] acpiphp: Slot [16] registered [ 0.418128] acpiphp: Slot [17] registered [ 0.419078] acpiphp: Slot [18] registered [ 0.420079] acpiphp: Slot [19] registered [ 0.422083] acpiphp: Slot [20] registered [ 0.423089] acpiphp: Slot [21] registered [ 0.424080] acpiphp: Slot [22] registered [ 0.425102] acpiphp: Slot [23] registered [ 0.426091] acpiphp: Slot [24] registered [ 0.427098] acpiphp: Slot [25] registered [ 0.428110] acpiphp: Slot [26] registered [ 0.430085] acpiphp: Slot [27] registered [ 0.431081] acpiphp: Slot [28] registered [ 0.432101] acpiphp: Slot [29] registered [ 0.433129] acpiphp: Slot [30] registered [ 0.434086] acpiphp: Slot [31] registered [ 0.436060] PCI host bridge to bus 0000:00 [ 0.437018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.439028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.440026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.443031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.445027] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.447030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.449193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.453848] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.457552] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.465376] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.469823] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.472023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.474018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.477024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.478472] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.480732] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.483044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.485670] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.488016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.497018] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.500018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.504316] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.511021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.517021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.532000] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.543989] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.548017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.567020] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.577744] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.580455] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.583389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.585350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.588291] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.592134] iommu: Default domain type: Passthrough [ 0.594481] SCSI subsystem initialized [ 0.599254] ACPI: bus type USB registered [ 0.600157] usbcore: registered new interface driver usbfs [ 0.603130] usbcore: registered new interface driver hub [ 0.605102] usbcore: registered new device driver usb [ 0.607190] pps_core: LinuxPPS API ver. 1 registered [ 0.608009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.611117] PTP clock support registered [ 0.614128] EDAC MC: Ver: 3.0.0 [ 0.616127] PCI: Using ACPI for IRQ routing [ 0.617695] NetLabel: Initializing [ 0.619015] NetLabel: domain hash size = 128 [ 0.620010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.622100] NetLabel: unlabeled traffic allowed by default [ 0.625112] vgaarb: loaded [ 0.626348] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.628016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.636364] clocksource: Switched to clocksource kvm-clock [ 0.745212] VFS: Disk quotas dquot_6.6.0 [ 0.746743] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.749070] *** VALIDATE ramfs *** [ 0.750326] *** VALIDATE hugetlbfs *** [ 0.751929] pnp: PnP ACPI init [ 0.754929] pnp: PnP ACPI: found 6 devices [ 0.772991] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.776566] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.778724] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.781028] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.783641] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.786161] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.788810] NET: Registered protocol family 2 [ 0.791380] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.796209] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.799902] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.805418] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.809176] TCP: Hash tables configured (established 65536 bind 65536) [ 0.812164] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.815232] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.818157] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.821036] NET: Registered protocol family 1 [ 0.832170] RPC: Registered named UNIX socket transport module. [ 0.834301] RPC: Registered udp transport module. [ 0.836229] RPC: Registered tcp transport module. [ 0.838137] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.840822] NET: Registered protocol family 44 [ 0.842370] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.844975] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.847522] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.849917] PCI: CLS 0 bytes, default 64 [ 0.851775] Unpacking initramfs... [ 2.287471] debug: unmapping init [mem 0xffff9878bcc64000-0xffff9878bffcffff] [ 2.291440] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.293751] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.296502] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.814118] Initialise system trusted keyrings [ 2.815872] Key type blacklist registered [ 2.818928] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.829749] zbud: loaded [ 2.832983] *** VALIDATE nfs *** [ 2.834184] *** VALIDATE nfs4 *** [ 2.835744] pstore: using deflate compression [ 2.839269] Platform Keyring initialized [ 2.944684] NET: Registered protocol family 38 [ 2.946354] Key type asymmetric registered [ 2.947701] Asymmetric key parser 'x509' registered [ 2.949480] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.952739] io scheduler mq-deadline registered [ 2.954627] io scheduler kyber registered [ 2.956474] io scheduler bfq registered [ 2.958215] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.961196] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.963726] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.966576] ACPI: Power Button [PWRF] [ 3.062096] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.155756] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.252601] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.281171] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.318287] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.323565] Non-volatile memory driver v1.3 [ 3.325275] Linux agpgart interface v0.103 [ 3.354976] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.357830] vda: detected capacity change from 0 to 68505600 [ 3.372535] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.375811] vdb: detected capacity change from 0 to 1073741824 [ 3.385673] libphy: Fixed MDIO Bus: probed [ 3.390645] usbcore: registered new interface driver usbserial_generic [ 3.392854] usbserial: USB Serial support registered for generic [ 3.395310] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.399615] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.401615] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.403938] mousedev: PS/2 mouse device common for all mice [ 3.406707] rtc_cmos 00:05: RTC can wake from S4 [ 3.409703] rtc_cmos 00:05: registered as rtc0 [ 3.411866] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.411952] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.414416] intel_pstate: CPU model not supported [ 3.421571] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.421893] hid: raw HID events driver (C) Jiri Kosina [ 3.426247] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.427300] usbcore: registered new interface driver usbhid [ 3.431459] usbhid: USB HID core driver [ 3.433105] drop_monitor: Initializing network drop monitor service [ 3.436254] Initializing XFRM netlink socket [ 3.438100] NET: Registered protocol family 10 [ 3.441056] Segment Routing with IPv6 [ 3.442647] NET: Registered protocol family 17 [ 3.447398] mpls_gso: MPLS GSO support [ 3.453770] RAS: Correctable Errors collector initialized. [ 3.455866] AVX version of gcm_enc/dec engaged. [ 3.457545] AES CTR mode by8 optimization enabled [ 3.552596] sched_clock: Marking stable (3552562155, 0)->(4486222633, -933660478) [ 3.556282] registered taskstats version 1 [ 3.558589] Loading compiled-in X.509 certificates [ 3.560458] zswap: loaded using pool lzo/zbud [ 3.588091] Key type big_key registered [ 3.604581] Key type encrypted registered [ 3.606438] ima: No TPM chip found, activating TPM-bypass! [ 3.608469] ima: Allocated hash algorithm: sha1 [ 3.609980] ima: No architecture policies found [ 3.611710] evm: Initialising EVM extended attributes: [ 3.613722] evm: security.selinux [ 3.614955] evm: security.ima [ 3.616140] evm: security.capability [ 3.617484] evm: HMAC attrs: 0x1 [ 3.620111] rtc_cmos 00:05: setting system clock to 2025-10-23 16:38:23 UTC (1761237503) [ 3.629556] debug: unmapping init [mem 0xffffffff8ae03000-0xffffffff8affffff] [ 3.632769] debug: unmapping init [mem 0xffffffff89b82000-0xffffffff89e58fff] [ 3.641147] Write protecting the kernel read-only data: 28672k [ 3.645250] debug: unmapping init [mem 0xffffffff88203000-0xffffffff883fffff] [ 3.647701] debug: unmapping init [mem 0xffffffff88b14000-0xffffffff88bfffff] [ 3.681626] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.689738] systemd[1]: Detected virtualization kvm. [ 3.691664] systemd[1]: Detected architecture x86-64. [ 3.693492] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.717867] systemd[1]: No hostname configured. [ 3.719387] systemd[1]: Set hostname to . [ 3.721135] random: systemd: uninitialized urandom read (16 bytes read) [ 3.723226] systemd[1]: Initializing machine ID from random generator. [ 3.771916] random: ln: uninitialized urandom read (6 bytes read) [ 3.875451] random: systemd: uninitialized urandom read (16 bytes read) [ 3.879417] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.883789] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.887177] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.580624] device-mapper: uevent: version 1.0.3 [ 4.584835] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.320033] random: fast init done [ 5.335922] virtio_net virtio0 ens2: renamed from eth0 [ 5.388863] scsi host0: ata_piix [ 5.392697] scsi host1: ata_piix [ 5.406063] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.409218] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.022294] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.125117] random: crng init done [ 10.126490] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.527623] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.718257] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.974072] SELinux: Disabled at runtime. [ 12.038394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.047527] systemd[1]: Detected virtualization kvm. [ 12.049382] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.618233] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.622067] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.631371] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.636252] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.643050] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.654950] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.660740] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 12.801205] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.253650] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.721064] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.737336] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.835550] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.878483] EDAC sbridge: Ver: 1.1.2 [ 15.187316] Key type dns_resolver registered [ 15.523592] NFS: Registering the id_resolver key type [ 15.525830] Key type id_resolver registered [ 15.528539] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-client login: [ 54.737769] libcfs: loading out-of-tree module taints kernel. [ 54.783859] Key type ._llcrypt registered [ 54.785167] Key type .llcrypt registered [ 55.536630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 55.546524] alg: No test for adler32 (adler32-zlib) [ 56.746086] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 57.564031] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 59.311146] Key type lgssc registered [ 60.772860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.767685] Lustre: Mounted lustre-client [ 190.454823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 206.157939] hrtimer: interrupt took 6927345 ns [ 207.645475] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.423317] Lustre: lustre-OST0000-osc-ffff98790981d000: disconnect after 24s idle [ 211.835107] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing yml_node [ 216.087730] Lustre: DEBUG MARKER: Client: 2.16.59.38 [ 218.742835] Lustre: DEBUG MARKER: MDS: 2.16.59.38 [ 221.524665] Lustre: DEBUG MARKER: OSS: 2.16.59.38 [ 223.062344] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Oct 23 12:42:01 EDT 2025 [ 238.396527] LustreError: 7909:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98790981d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 238.412182] LustreError: 7909:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 238.464184] Lustre: Unmounted lustre-client [ 256.134814] Key type lgssc unregistered [ 256.375661] LNet: 8354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.388547] LNetError: 8354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 257.451215] LNet: Removed LNI 192.168.202.31@tcp [ 258.367312] Key type .llcrypt unregistered [ 258.370326] Key type ._llcrypt unregistered [ 283.667715] Key type ._llcrypt registered [ 283.670561] Key type .llcrypt registered [ 283.925098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 283.956671] alg: No test for adler32 (adler32-zlib) [ 284.906275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 284.977344] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 284.992539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.240321] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 289.332347] LNet: 9220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.337119] LNetError: 9220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.353876] LNet: Removed LNI 192.168.202.31@tcp [ 290.049486] Key type .llcrypt unregistered [ 290.130862] Key type ._llcrypt unregistered [ 296.803295] Key type ._llcrypt registered [ 296.806053] Key type .llcrypt registered [ 298.474191] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:43:16 (1761237796) [ 298.988769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 299.002037] alg: No test for adler32 (adler32-zlib) [ 299.922396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.005826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.018894] LNet: 9635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.052239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.103615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.139477] LNet: 9642:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 310.073727] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:43:27 (1761237807) [ 310.207437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.268573] LNet: 10214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.327222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.562563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 319.267698] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:43:37 (1761237817) [ 319.418283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.464347] LNet: 10803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.508945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.601358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 327.921534] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:43:46 (1761237826) [ 327.986690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.998873] LNet: 11383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.029521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.127461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 334.888939] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:43:53 (1761237833) [ 335.062260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.105439] LNet: 11974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.128913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.181498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 335.454102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 342.648496] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:44:00 (1761237840) [ 342.802680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.885490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.993768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 343.578896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 344.040815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 351.065308] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:44:09 (1761237849) [ 351.145179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.191152] LNet: 13156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.195775] LNet: 13156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 351.240164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.364911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 357.713234] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:44:16 (1761237856) [ 357.981200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.035716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.110910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 358.162656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 358.674523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 358.707449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.186039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 359.216524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.569246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 359.606525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.114181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 360.174990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.504501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 360.578294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.022992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 361.068931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.470778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 361.517538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 361.772965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 361.801550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.106191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 362.134330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 362.552921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 362.582960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 363.083628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 363.123580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.384298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 363.475303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 369.640889] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:44:28 (1761237868) [ 369.919330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.937779] LNet: 14432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.942668] LNet: 14432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.971211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.054189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 370.097260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 376.282762] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:44:34 (1761237874) [ 376.407708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.458364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.561289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 376.598842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 382.422513] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:44:41 (1761237881) [ 382.688436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.710976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.823570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 382.853400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 388.471705] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:44:47 (1761237887) [ 388.575195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.625473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.749248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 388.817470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 395.450274] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:44:54 (1761237894) [ 395.662579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.730633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.776602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 395.817633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 402.897193] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:45:01 (1761237901) [ 403.101306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.129869] LNet: 17362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.133311] LNet: 17362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 403.146516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.238547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 403.289702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 409.019971] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:45:07 (1761237907) [ 409.129841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.212716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.358138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.697887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 415.769532] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:45:14 (1761237914) [ 415.863418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.909078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.110634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 416.363153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 423.317733] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:45:21 (1761237921) [ 423.725386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.759354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.875950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 424.080359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.254890] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:45:29 (1761237929) [ 431.359950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.414886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.878968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.234737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.426799] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:45:37 (1761237937) [ 439.545180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.595435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.694175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.821555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.394945] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:45:45 (1761237945) [ 446.596520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.656338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.763345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.816401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.101842] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:45:52 (1761237952) [ 454.186658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.229844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.319408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 460.601720] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:45:59 (1761237959) [ 460.903580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.952895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.034795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 466.027227] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:46:04 (1761237964) [ 466.259960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.287881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.341590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 466.707389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 471.780031] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:46:10 (1761237970) [ 471.935812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.950376] LNet: 23388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.960380] LNet: 23388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 472.000899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.057497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 472.103240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 478.366741] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:46:17 (1761237977) [ 478.669438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.734404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.821295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 478.859980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 485.957031] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:46:24 (1761237984) [ 486.074101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.144682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.260642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 486.298611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 492.912354] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:46:31 (1761237991) [ 493.196663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.256825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.322738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 493.347268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 499.519737] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:46:38 (1761237998) [ 499.820410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.878401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.950890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 500.045541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 500.058579] LNetError: 25747:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 500.097706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 500.123592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 506.471891] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:46:45 (1761238005) [ 506.816302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.868370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.973555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NvMaHSK7tD [ 507.061672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NvMaHSK7tD [ 507.129448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.NvMaHSK7tD [ 513.588141] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:46:52 (1761238012) [ 513.852483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.958735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.079750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 514.165544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 514.205345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 520.947538] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:46:59 (1761238019) [ 521.115525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.165156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.243264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 523.426254] LNet: Message forwarding will be enabled in 2s [ 527.519165] LNet: Message forwarding enabled [ 528.072342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 528.124213] LNet: Message forwarding will stop in 2s to 6s [ 532.640925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 533.338499] LNet: Message forwarding disabled [ 534.620167] LNet: Message forwarding will be enabled in 2s [ 538.472615] LNet: Message forwarding enabled [ 542.095260] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:47:20 (1761238040) [ 542.223064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.107106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.243996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 551.147710] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:47:29 (1761238049) [ 551.552604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.590455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.631027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 551.659494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 551.687768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 551.727677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 551.760789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 551.800188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 551.834410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 551.880964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 551.948370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 551.988484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 552.047168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 552.119652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 552.189231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 552.234963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 552.268661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 552.323794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 552.400921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 552.461605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 552.501186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 552.559330] LNetError: 28828:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 558.978816] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:47:37 (1761238057) [ 559.154885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.205753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.289609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.842759] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:47:44 (1761238064) [ 567.679705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg231-server [ 569.480350] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:47:48 (1761238068) [ 569.919781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.987498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.058854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.095849] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 571.976254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[32-36]@tcp [ 572.015334] LNet: 30541:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 572.115934] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 572.141982] LNetError: 30333:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.32@tcp has gone from up to down [ 573.097730] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.34@tcp is being used as a gateway but routing feature is not turned on [ 573.129888] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 574.119591] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 574.141747] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 576.183835] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 576.207797] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 579.812897] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:47:58 (1761238078) [ 580.290787] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.32@tcp is being used as a gateway but routing feature is not turned on [ 580.305043] LNetError: 30333:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 581.773636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg231-server [ 583.329920] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:48:02 (1761238082) [ 583.646514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.675564] LNetError: 31475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.753105] LNet: Removed LNI 192.168.202.31@tcp [ 584.802979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.875143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 584.921458] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 585.626036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[32-37/2]@tcp [ 585.635891] LNet: 31590:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 585.640314] LNet: 31590:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 585.660755] LNetError: 31479:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.34@tcp has gone from up to down [ 585.664325] LNetError: 31479:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 585.666753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[32-37/2]@tcp [ 592.072504] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:48:10 (1761238090) [ 592.365627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.401521] LNetError: 32182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.423773] LNet: Removed LNI 192.168.202.31@tcp [ 592.458586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.594825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 592.687157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.735251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.824718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.889660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.933566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.001250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 593.106882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.167731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.279708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.398448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.437421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.490845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 593.587889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.638861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.697883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.799549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.862508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.918947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 594.007772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.092653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.191649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.281751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.363419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.422457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 594.560136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.593042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.747190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.857837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.895415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.936314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 594.989787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.026300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.079208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 601.386581] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:48:19 (1761238099) [ 603.320640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg231-server [ 604.912878] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:48:23 (1761238103) [ 606.655916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg231-server [ 608.418754] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:48:26 (1761238106) [ 608.547139] LNet: 33579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.558721] LNet: 33579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 24 previous similar messages [ 609.312078] Key type .llcrypt unregistered [ 609.316978] Key type ._llcrypt unregistered [ 614.695548] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 614.920288] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 614.924597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 614.932989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 617.464771] Key type ._llcrypt registered [ 617.466423] Key type .llcrypt registered [ 617.698712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.718807] alg: No test for adler32 (adler32-zlib) [ 618.616482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.673611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.710418] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 619.355291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 619.377325] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 621.343615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 621.374943] LNet: Removed LNI 10.1.2.3@tcp [ 621.537560] LNet: 34294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.549952] LNetError: 34294:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.574915] LNet: Removed LNI 192.168.202.31@tcp [ 622.295625] Key type .llcrypt unregistered [ 622.298246] Key type ._llcrypt unregistered [ 632.333279] Key type ._llcrypt registered [ 632.335188] Key type .llcrypt registered [ 633.495859] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:48:52 (1761238132) [ 633.916474] Key type .llcrypt unregistered [ 633.918142] Key type ._llcrypt unregistered [ 638.310213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 638.442710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 638.444940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 638.449252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 641.063875] Key type ._llcrypt registered [ 641.065979] Key type .llcrypt registered [ 641.241455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 641.278759] alg: No test for adler32 (adler32-zlib) [ 642.216430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.283732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.308462] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 642.966540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 642.980362] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 645.548289] LNet: 35774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.561168] LNetError: 35774:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.568067] LNet: Removed LNI 192.168.202.31@tcp [ 645.593801] LNet: Removed LNI 10.1.2.3@tcp [ 646.192262] Key type .llcrypt unregistered [ 646.198271] Key type ._llcrypt unregistered [ 656.406906] Key type ._llcrypt registered [ 656.410198] Key type .llcrypt registered [ 657.728225] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:49:16 (1761238156) [ 658.377800] Key type .llcrypt unregistered [ 658.381709] Key type ._llcrypt unregistered [ 664.104834] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 664.276963] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 664.280134] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 664.282914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 667.063857] Key type ._llcrypt registered [ 667.071495] Key type .llcrypt registered [ 667.451871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 667.484190] alg: No test for adler32 (adler32-zlib) [ 668.510471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.645875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 668.688985] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 671.339419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 671.368040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.185683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 674.210120] LNet: Removed LNI 10.1.2.3@tcp [ 674.231963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 674.288451] LNet: Removed LNI 10.1.2.31@tcp [ 674.310911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 674.389559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 675.422385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 675.464888] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 676.892206] LNet: 37675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.898632] LNetError: 37675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 676.915487] LNet: Removed LNI 10.1.2.3@tcp [ 676.924529] LNet: Removed LNI 10.1.2.31@tcp [ 677.693653] Key type .llcrypt unregistered [ 677.695346] Key type ._llcrypt unregistered [ 692.056591] Key type ._llcrypt registered [ 692.060678] Key type .llcrypt registered [ 693.810814] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:49:51 (1761238191) [ 694.828615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.847902] alg: No test for adler32 (adler32-zlib) [ 695.781243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.851728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.878311] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 696.814687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.832290] LNet: 38623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.842484] LNetError: 38623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.905923] LNet: Removed LNI 192.168.202.31@tcp [ 696.919912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.974575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.31@tcp [ 697.043554] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 698.026507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.31@tcp [ 698.056301] LNet: Removed LNI 192.168.202.31@tcp [ 698.170159] LNet: 38755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.863688] Key type .llcrypt unregistered [ 698.865240] Key type ._llcrypt unregistered [ 709.488338] Key type ._llcrypt registered [ 709.490211] Key type .llcrypt registered [ 710.790414] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:50:09 (1761238209) [ 712.330368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg231-server [ 714.274820] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:50:12 (1761238212) [ 714.644355] Key type .llcrypt unregistered [ 714.646468] Key type ._llcrypt unregistered [ 720.256903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 720.395701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 720.423168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 720.437706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.227494] Key type ._llcrypt registered [ 723.229884] Key type .llcrypt registered [ 723.516259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.526027] alg: No test for adler32 (adler32-zlib) [ 724.497562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.603393] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 725.381910] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 727.821574] LNet: 40580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.828637] LNetError: 40580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.834251] LNet: Removed LNI 192.168.202.31@tcp1 [ 727.846791] LNet: Removed LNI 10.1.2.3@tcp2 [ 728.397860] Key type .llcrypt unregistered [ 728.399830] Key type ._llcrypt unregistered [ 740.485405] Key type ._llcrypt registered [ 740.487375] Key type .llcrypt registered [ 741.885798] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:50:40 (1761238240) [ 742.351516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 742.373300] alg: No test for adler32 (adler32-zlib) [ 743.283418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.346295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.393294] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 744.176512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.207200] LNet: 41542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.216349] LNetError: 41542:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.226788] LNet: Removed LNI 192.168.202.31@tcp [ 744.249270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.313486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 744.333298] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 751.091670] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:50:49 (1761238249) [ 751.157529] LNet: 42125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.161932] LNetError: 42125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.181395] LNet: Removed LNI 192.168.202.31@tcp [ 751.770419] Key type .llcrypt unregistered [ 751.774670] Key type ._llcrypt unregistered [ 756.825993] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.933475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.937061] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.940206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.539268] Key type ._llcrypt registered [ 759.546631] Key type .llcrypt registered [ 759.815088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.839784] alg: No test for adler32 (adler32-zlib) [ 760.718786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.768818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.784770] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 761.418866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 761.459585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.379765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.403787] LNet: 42862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.408946] LNetError: 42862:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.413737] LNet: Removed LNI 192.168.202.31@tcp [ 763.449722] LNet: Removed LNI 10.1.2.3@tcp [ 763.470690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.518488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 763.540419] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 763.544443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.872831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.889778] LNet: Removed LNI 192.168.202.31@tcp [ 763.902721] LNet: Removed LNI 10.1.2.3@tcp [ 763.926257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.005670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 764.042138] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 764.045868] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.357197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 764.380284] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 765.204403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.227344] LNet: 43048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.231058] LNet: 43048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 765.249994] LNetError: 43048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.253404] LNetError: 43048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 765.257029] LNet: Removed LNI 192.168.202.31@tcp [ 765.258872] LNet: Removed LNI 10.1.2.3@tcp [ 765.284555] LNet: Removed LNI 192.168.202.31@tcp2 [ 765.305854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.388187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 765.422879] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 765.428116] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.437334] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 771.569067] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:51:10 (1761238270) [ 771.894242] LNet: 43644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.897463] LNetError: 43644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.900503] LNetError: 43644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 771.918082] LNet: Removed LNI 192.168.202.31@tcp [ 771.926183] LNet: Removed LNI 10.1.2.3@tcp [ 771.951109] LNet: Removed LNI 192.168.202.31@tcp2 [ 772.702438] Key type .llcrypt unregistered [ 772.704621] Key type ._llcrypt unregistered [ 778.077068] Key type ._llcrypt registered [ 778.079279] Key type .llcrypt registered [ 778.323229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.343720] alg: No test for adler32 (adler32-zlib) [ 779.295652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.340572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.354302] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 780.046302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.072159] LNet: 44033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.076152] LNetError: 44033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.092124] LNet: Removed LNI 192.168.202.31@tcp [ 780.122135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.206963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 780.263370] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 780.731677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.751994] LNet: 44063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.758387] LNetError: 44063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.779419] LNet: Removed LNI 192.168.202.31@tcp [ 780.800949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.838567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 780.860632] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 781.216428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.256968] LNet: Removed LNI 192.168.202.31@tcp [ 781.274523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.324618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.350177] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 781.739912] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.795771] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.798292] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.801489] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.148870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.163765] LNet: 44166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.168104] LNet: 44166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 784.174987] LNetError: 44166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.179579] LNetError: 44166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.193318] LNet: Removed LNI 192.168.202.31@tcp [ 784.223114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.308550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 784.334662] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 790.346219] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:51:29 (1761238289) [ 790.578309] LNet: 44763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.589217] LNetError: 44763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.610192] LNet: Removed LNI 192.168.202.31@tcp [ 791.271950] Key type .llcrypt unregistered [ 791.274837] Key type ._llcrypt unregistered [ 796.380307] Key type ._llcrypt registered [ 796.381976] Key type .llcrypt registered [ 796.542302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.554889] alg: No test for adler32 (adler32-zlib) [ 797.490685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.538710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.564284] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 798.283866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.314473] LNet: 45153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.318939] LNetError: 45153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.331797] LNet: Removed LNI 192.168.202.31@tcp [ 798.364217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.437582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 798.459725] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 805.245081] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:51:43 (1761238303) [ 805.345647] LNet: 45737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.362094] LNetError: 45737:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.399689] LNet: Removed LNI 192.168.202.31@tcp [ 806.149753] Key type .llcrypt unregistered [ 806.153905] Key type ._llcrypt unregistered [ 811.797876] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.836671] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.839696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.842276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 813.332100] Key type ._llcrypt registered [ 813.333723] Key type .llcrypt registered [ 813.505202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.521863] alg: No test for adler32 (adler32-zlib) [ 814.418562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.468894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.492220] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 815.237322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.255435] LNet: 46176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.265594] LNetError: 46176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.283057] LNet: Removed LNI 192.168.202.31@tcp [ 815.309983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.368862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 815.404515] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 821.412591] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:52:00 (1761238320) [ 821.493224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.515558] LNet: 46766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.519862] LNetError: 46766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.533304] LNet: Removed LNI 192.168.202.31@tcp [ 821.548408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.642787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 826.699652] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:52:05 (1761238325) [ 827.024116] LNet: 47355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.575425] Key type .llcrypt unregistered [ 827.577075] Key type ._llcrypt unregistered [ 832.934520] Key type ._llcrypt registered [ 832.937976] Key type .llcrypt registered [ 833.116306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.132475] alg: No test for adler32 (adler32-zlib) [ 834.055456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.104303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.125499] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 835.069925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.085494] LNet: 47746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.091422] LNetError: 47746:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.104652] LNet: Removed LNI 192.168.202.31@tcp [ 835.139593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.184279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 835.211753] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 841.465991] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:52:20 (1761238340) [ 841.772036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.814368] LNet: 48339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.822203] LNetError: 48339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.854036] LNet: Removed LNI 192.168.202.31@tcp [ 841.874389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.956375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 848.638737] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:52:26 (1761238346) [ 848.752460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.787838] LNet: 48914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.832360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.902380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.948157] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 849.675503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.31 [ 849.700715] LNet: Removed LNI 192.168.202.31@tcp [ 855.927280] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:52:34 (1761238354) [ 856.066688] LNet: 49619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.071531] LNetError: 49619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.081874] LNet: Removed LNI 192.168.202.31@tcp [ 856.644591] Key type .llcrypt unregistered [ 856.646198] Key type ._llcrypt unregistered [ 863.872212] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 864.198930] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.201140] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.227868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.940791] Key type ._llcrypt registered [ 866.943317] Key type .llcrypt registered [ 867.106452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.133837] alg: No test for adler32 (adler32-zlib) [ 868.551266] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.329560] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:52:54 (1761238374) [ 876.602708] LNet: 50521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.614685] LNetError: 50521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 876.649799] LNet: Removed LNI 10.1.2.3@tcp [ 877.523514] Key type .llcrypt unregistered [ 877.529307] Key type ._llcrypt unregistered [ 885.188861] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.543172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.545847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.565761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.560102] Key type ._llcrypt registered [ 888.562504] Key type .llcrypt registered [ 888.810059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.828274] alg: No test for adler32 (adler32-zlib) [ 889.805841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.697688] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:53:14 (1761238394) [ 896.018509] LNet: 51411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.023788] LNetError: 51411:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 896.037604] LNet: Removed LNI 10.1.2.3@tcp [ 896.638327] Key type .llcrypt unregistered [ 896.642071] Key type ._llcrypt unregistered [ 902.673943] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 903.023928] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 903.047352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 903.060461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 905.818613] Key type ._llcrypt registered [ 905.820242] Key type .llcrypt registered [ 905.960725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.980992] alg: No test for adler32 (adler32-zlib) [ 906.952287] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 914.078898] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:53:32 (1761238412) [ 914.339787] LNet: 52301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.346687] LNetError: 52301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.360279] LNet: Removed LNI 10.1.2.3@tcp [ 915.063474] Key type .llcrypt unregistered [ 915.065175] Key type ._llcrypt unregistered [ 921.050367] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 921.288549] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 921.292823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 921.301349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 924.083533] Key type ._llcrypt registered [ 924.085088] Key type .llcrypt registered [ 924.262680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 924.275686] alg: No test for adler32 (adler32-zlib) [ 925.189980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.253225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 933.722716] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:53:52 (1761238432) [ 934.386327] LNet: 53525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.392239] LNetError: 53525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 934.413990] LNet: Removed LNI 10.1.2.3@tcp [ 935.286098] Key type .llcrypt unregistered [ 935.291458] Key type ._llcrypt unregistered [ 941.750432] Key type ._llcrypt registered [ 941.751895] Key type .llcrypt registered [ 942.088762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.124703] alg: No test for adler32 (adler32-zlib) [ 943.198647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 943.245721] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 950.282315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 951.210825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_interrupt -r 1 [ 951.252406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_interrupt -r 1 [ 951.295644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 951.314541] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 951.623899] LNet: 53950:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 951.688349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.718670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.758982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 952.280717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_dropped -r 1 [ 952.328754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_dropped -r 1 [ 952.351317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 952.378896] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 952.669658] LNet: 54048:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.739791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.772456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 952.807120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 953.350670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_aborted -r 1 [ 953.377443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_aborted -r 1 [ 953.410819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 953.431162] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 953.687030] LNet: 54145:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 953.775865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.802939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.825544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 954.340194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_no_route -r 1 [ 954.362427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_no_route -r 1 [ 954.406794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 954.765520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.810471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.834428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 955.341829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_timeout -r 1 [ 955.383700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_timeout -r 1 [ 955.414992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 955.439970] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 955.449479] LNet: Skipped 1 previous similar message [ 955.770382] LNet: 54339:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 955.778712] LNet: 54339:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 955.857504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.889377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.916728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 956.474634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_error -r 1 [ 956.530937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_error -r 1 [ 956.573280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 957.006593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 957.045319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 957.095275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 957.459571] LNet: 54473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.474173] LNetError: 54473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.504803] LNet: Removed LNI 192.168.202.31@tcp [ 957.915592] Key type .llcrypt unregistered [ 957.917724] Key type ._llcrypt unregistered [ 973.484144] Key type ._llcrypt registered [ 973.485861] Key type .llcrypt registered [ 975.006477] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:54:33 (1761238473) [ 976.062580] Key type .llcrypt unregistered [ 976.064195] Key type ._llcrypt unregistered [ 981.597194] Key type ._llcrypt registered [ 981.598850] Key type .llcrypt registered [ 981.824825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.841499] alg: No test for adler32 (adler32-zlib) [ 982.742949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 982.782660] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 990.749279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1000.335379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.432381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1000.482824] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1002.249129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_interrupt -r 1 [ 1002.291430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 1 [ 1002.355263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_interrupt -r 1 [ 1002.408334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 1 [ 1002.470780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_interrupt -r 1 [ 1002.525666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 1 [ 1002.601259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_interrupt -r 1 [ 1002.693975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 1 [ 1002.735533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1002.766949] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1002.801430] LNet: 2 local NIs in recovery (showing 2): 192.168.202.31@tcp, 192.168.202.31@tcp1 [ 1003.199343] LNet: 55924:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.273311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.304105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.338676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1003.863162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_dropped -r 1 [ 1003.919781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 1 [ 1003.988638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_dropped -r 1 [ 1004.033606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 1 [ 1004.076057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_dropped -r 1 [ 1004.143873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 1 [ 1004.207709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_dropped -r 1 [ 1004.265704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 1 [ 1004.327500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1004.374193] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1004.388179] LNet: Skipped 2 previous similar messages [ 1004.788535] LNet: 56033:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1004.860945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1004.897618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1004.930343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1005.595280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_aborted -r 1 [ 1005.620623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 1 [ 1005.667413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_aborted -r 1 [ 1005.712527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 1 [ 1005.749628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_aborted -r 1 [ 1005.789892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 1 [ 1005.821532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_aborted -r 1 [ 1005.852539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 1 [ 1005.901623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1005.923097] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1005.928387] LNet: Skipped 2 previous similar messages [ 1006.310485] LNet: 56142:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1006.381627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.424934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.474636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1007.113958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_no_route -r 1 [ 1007.149215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 1 [ 1007.186244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_no_route -r 1 [ 1007.218993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 1 [ 1007.274131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_no_route -r 1 [ 1007.337214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 1 [ 1007.408079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_no_route -r 1 [ 1007.470699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 1 [ 1007.514667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1008.007963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.060588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.114620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1008.794720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_timeout -r 1 [ 1008.829601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 1 [ 1008.899217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_timeout -r 1 [ 1008.952878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 1 [ 1008.995680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_timeout -r 1 [ 1009.033684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 1 [ 1009.064412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_timeout -r 1 [ 1009.098884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 1 [ 1009.128423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1009.155469] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1009.158949] LNet: Skipped 5 previous similar messages [ 1009.538364] LNet: 56361:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1009.549387] LNet: 56361:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1009.624297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1009.674265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1009.700658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1010.389235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_error -r 1 [ 1010.430498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_error -r 1 [ 1010.474504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_error -r 1 [ 1010.533089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_error -r 1 [ 1010.574424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_error -r 1 [ 1010.639885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_error -r 1 [ 1010.711780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_error -r 1 [ 1010.767957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_error -r 1 [ 1010.833364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1011.289713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.329976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.363768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1016.731553] LNet: 56541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.737549] LNetError: 56541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.759872] LNet: Removed LNI 192.168.202.31@tcp [ 1017.765086] LNetError: 56541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.813993] LNet: Removed LNI 192.168.202.31@tcp1 [ 1019.427977] Key type .llcrypt unregistered [ 1019.432361] Key type ._llcrypt unregistered [ 1036.565557] Key type ._llcrypt registered [ 1036.567914] Key type .llcrypt registered [ 1038.251267] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:55:36 (1761238536) [ 1039.108320] Key type .llcrypt unregistered [ 1039.109947] Key type ._llcrypt unregistered [ 1045.333309] Key type ._llcrypt registered [ 1045.334920] Key type .llcrypt registered [ 1045.593634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.615359] alg: No test for adler32 (adler32-zlib) [ 1046.585994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1046.647975] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1054.697659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1055.780378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_dropped -r 1 [ 1055.841117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_dropped -r 1 [ 1055.910738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1055.954108] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1056.245807] LNet: 57779:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.335996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.374786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.416271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.987598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_error -r 1 [ 1057.038917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_error -r 1 [ 1057.078373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1057.093906] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1057.446718] LNet: 57876:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.519866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.547830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.584209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1058.235674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_timeout -r 1 [ 1058.283937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_timeout -r 1 [ 1058.339815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1058.372464] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1058.742554] LNet: 57973:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1058.836591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.877541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.915524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.205612] LNet: 58009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.216560] LNetError: 58009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.251493] LNet: Removed LNI 192.168.202.31@tcp [ 1059.770514] Key type .llcrypt unregistered [ 1059.772306] Key type ._llcrypt unregistered [ 1076.222962] Key type ._llcrypt registered [ 1076.224506] Key type .llcrypt registered [ 1077.692845] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:56:15 (1761238575) [ 1078.625459] Key type .llcrypt unregistered [ 1078.630980] Key type ._llcrypt unregistered [ 1085.460678] Key type ._llcrypt registered [ 1085.461847] Key type .llcrypt registered [ 1085.705707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.735868] alg: No test for adler32 (adler32-zlib) [ 1086.714343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1086.754776] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1093.731148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1102.869506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.948634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1102.976300] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1104.858510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_dropped -r 1 [ 1104.887826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 1 [ 1104.917568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_dropped -r 1 [ 1104.950608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 1 [ 1104.975652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_dropped -r 1 [ 1105.006514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 1 [ 1105.032435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_dropped -r 1 [ 1105.080154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 1 [ 1105.112985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1105.131994] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1105.144090] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.131@tcp [ 1105.475666] LNet: 59476:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.542674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.573295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.607858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.639098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.214913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_error -r 1 [ 1106.255953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_error -r 1 [ 1106.318800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_error -r 1 [ 1106.396761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_error -r 1 [ 1106.473422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_error -r 1 [ 1106.532840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_error -r 1 [ 1106.572462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_error -r 1 [ 1106.606791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_error -r 1 [ 1106.660281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1106.691630] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1106.698752] LNet: Skipped 2 previous similar messages [ 1107.066560] LNet: 59587:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.153235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.193798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.234775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.278401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.803918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_timeout -r 1 [ 1107.850986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_timeout -r 1 [ 1107.912505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_timeout -r 1 [ 1107.966323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_timeout -r 1 [ 1108.006741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_timeout -r 1 [ 1108.035972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_timeout -r 1 [ 1108.070932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_timeout -r 1 [ 1108.118859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_timeout -r 1 [ 1108.162910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.131@tcp [ 1108.181867] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1108.586382] LNet: 59698:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.660412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.695384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.730172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.756637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.346883] LNet: 59771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.354715] LNetError: 59771:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.360288] LNet: Removed LNI 192.168.202.31@tcp [ 1113.368763] LNet: Removed LNI 192.168.202.31@tcp1 [ 1113.754224] Key type .llcrypt unregistered [ 1113.755460] Key type ._llcrypt unregistered [ 1129.274719] Key type ._llcrypt registered [ 1129.280332] Key type .llcrypt registered [ 1130.440779] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:57:09 (1761238629) [ 1131.098379] Key type .llcrypt unregistered [ 1131.101820] Key type ._llcrypt unregistered [ 1136.707018] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1136.946153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1136.949394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1136.951876] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1139.715749] Key type ._llcrypt registered [ 1139.717476] Key type .llcrypt registered [ 1139.923980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.944479] alg: No test for adler32 (adler32-zlib) [ 1140.920891] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1140.993209] LNet: 60853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.003302] LNetError: 60853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.024784] LNet: Removed LNI 192.168.202.31@tcp [ 1141.689517] Key type .llcrypt unregistered [ 1141.691402] Key type ._llcrypt unregistered [ 1147.636780] Key type ._llcrypt registered [ 1147.638496] Key type .llcrypt registered [ 1147.855773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.886579] alg: No test for adler32 (adler32-zlib) [ 1148.851910] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1148.854188] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1148.885955] LNet: 61123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.890341] LNetError: 61123:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.893977] LNet: Removed LNI 192.168.202.31@tcp [ 1148.906535] LNet: Removed LNI 10.1.2.3@tcp1 [ 1149.534841] Key type .llcrypt unregistered [ 1149.536690] Key type ._llcrypt unregistered [ 1155.196116] Key type ._llcrypt registered [ 1155.197803] Key type .llcrypt registered [ 1155.393613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1155.413382] alg: No test for adler32 (adler32-zlib) [ 1156.418182] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1156.420713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1156.478895] LNet: 61392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.487812] LNetError: 61392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.491737] LNet: Removed LNI 192.168.202.31@tcp [ 1156.510924] LNet: Removed LNI 10.1.2.3@tcp [ 1157.386741] Key type .llcrypt unregistered [ 1157.392938] Key type ._llcrypt unregistered [ 1163.888380] Key type ._llcrypt registered [ 1163.890022] Key type .llcrypt registered [ 1164.115518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.126836] alg: No test for adler32 (adler32-zlib) [ 1165.206730] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1165.210709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1165.281371] LNet: 61663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.288769] LNetError: 61663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.292519] LNet: Removed LNI 192.168.202.31@tcp [ 1165.320877] LNet: Removed LNI 10.1.2.3@tcp [ 1166.192706] Key type .llcrypt unregistered [ 1166.195398] Key type ._llcrypt unregistered [ 1171.759371] Key type ._llcrypt registered [ 1171.761032] Key type .llcrypt registered [ 1172.001781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.065694] alg: No test for adler32 (adler32-zlib) [ 1173.078514] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1173.081243] LNetError: 61924:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1173.086110] LNetError: Error -100 starting up LNI tcp [ 1173.094774] LNet: Removed LNI 192.168.202.31@tcp [ 1173.704790] Key type .llcrypt unregistered [ 1173.706344] Key type ._llcrypt unregistered [ 1184.921448] Key type ._llcrypt registered [ 1184.923118] Key type .llcrypt registered [ 1186.292760] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:58:04 (1761238684) [ 1187.162596] Key type .llcrypt unregistered [ 1187.167720] Key type ._llcrypt unregistered [ 1192.274127] Key type ._llcrypt registered [ 1192.277745] Key type .llcrypt registered [ 1192.571068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.611628] alg: No test for adler32 (adler32-zlib) [ 1193.545633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1193.587313] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1200.294002] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1201.066116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e network_timeout -r 1 [ 1201.100440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e network_timeout -r 1 [ 1201.128616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1201.146833] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1201.415781] LNet: 63165:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.472855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.504037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.739799] LNet: 63199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.751086] LNetError: 63199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.771837] LNet: Removed LNI 192.168.202.31@tcp [ 1202.110715] Key type .llcrypt unregistered [ 1202.112879] Key type ._llcrypt unregistered [ 1214.662571] Key type ._llcrypt registered [ 1214.667844] Key type .llcrypt registered [ 1214.877533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1214.895950] alg: No test for adler32 (adler32-zlib) [ 1215.782621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1215.817951] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1221.488435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1228.213688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.284140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1228.341544] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1230.132276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e network_timeout -r 1 [ 1230.202502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e network_timeout -r 1 [ 1230.259581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e network_timeout -r 1 [ 1230.298395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e network_timeout -r 1 [ 1230.362153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e network_timeout -r 1 [ 1230.428321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e network_timeout -r 1 [ 1230.470253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e network_timeout -r 1 [ 1230.493978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e network_timeout -r 1 [ 1230.539147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1230.570754] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1230.925536] LNet: 64061:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.998815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.065945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.791702] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1235.081241] LNet: 64130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.084973] LNetError: 64130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.090611] LNet: Removed LNI 192.168.202.31@tcp [ 1235.099373] LNet: Removed LNI 192.168.202.31@tcp1 [ 1235.556717] Key type .llcrypt unregistered [ 1235.558359] Key type ._llcrypt unregistered [ 1247.074272] Key type ._llcrypt registered [ 1247.076335] Key type .llcrypt registered [ 1248.206202] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:59:06 (1761238746) [ 1248.465272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.490185] alg: No test for adler32 (adler32-zlib) [ 1249.344501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.405912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1249.424167] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1249.940372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1249.957248] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1250.686419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1250.758994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1250.832811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1250.871165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1250.906800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1250.935832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1250.966772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.31@tcp [ 1250.981939] LNetError: 65220:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.202.31@tcp to recovery [ 1251.423362] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1251.426184] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1253.473110] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1254.352414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1257.567555] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1257.789391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1257.839914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1257.914696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.942813] LNet: 65327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.951959] LNetError: 65327:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.957984] LNet: Removed LNI 192.168.202.31@tcp [ 1257.970140] LNet: Removed LNI 192.168.202.31@tcp1 [ 1257.997408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.077851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1258.131508] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1258.788234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.808762] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1259.565354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1259.606237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.668075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1259.709436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1259.740629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1259.785589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1259.825313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1259.906825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.31@tcp [ 1259.926547] LNetError: 65594:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.202.31@tcp to recovery [ 1260.129135] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1263.313992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1266.271451] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1266.278088] LNet: Skipped 1 previous similar message [ 1271.286213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1271.333651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1271.359657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.034986] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:59:37 (1761238777) [ 1279.340707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1279.404844] LNet: 66312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.429126] LNetError: 66312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.439182] LNetError: 66312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1279.454316] LNet: Removed LNI 192.168.202.31@tcp [ 1279.485080] LNet: Removed LNI 192.168.202.31@tcp1 [ 1279.522028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.624508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1279.657342] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1280.613578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1280.647590] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1281.665346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1281.742752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1281.791546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1281.829482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1281.860030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1281.909658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1281.934991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.31@tcp --health 0 [ 1282.011332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1282.658978] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.31@tcp [ 1282.660709] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1282.693531] LNet: Skipped 1 previous similar message [ 1296.487122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1296.688703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1296.726931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.757442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.31@tcp --health 500 [ 1296.895397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1302.576253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1302.636103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.667434] LNet: 66782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.676950] LNetError: 66782:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.681537] LNetError: 66782:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1302.686472] LNet: Removed LNI 192.168.202.31@tcp [ 1302.699774] LNet: Removed LNI 192.168.202.31@tcp1 [ 1302.727669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.785891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1302.805572] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1303.618323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1303.640379] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1304.461691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1304.514887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.621744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1304.653805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1304.692413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1304.740287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1304.791036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1304.815271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.31@tcp --health 0 [ 1305.823495] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1305.837409] LNet: Skipped 3 previous similar messages [ 1306.186892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.282654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.325335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1315.363750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.302890] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:00:20 (1761238820) [ 1321.697333] LNet: 67810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.704661] LNetError: 67810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.708588] LNetError: 67810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1321.713556] LNet: Removed LNI 192.168.202.31@tcp [ 1321.730844] LNet: Removed LNI 192.168.202.31@tcp1 [ 1322.235139] Key type .llcrypt unregistered [ 1322.238746] Key type ._llcrypt unregistered [ 1326.748226] Key type ._llcrypt registered [ 1326.753364] Key type .llcrypt registered [ 1326.973373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1326.987949] alg: No test for adler32 (adler32-zlib) [ 1327.973961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1328.012549] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1335.725922] Lustre: DEBUG MARKER: Initial discovery [ 1335.851145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.131@tcp [ 1338.162376] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1338.247475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1338.289048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.131@tcp [ 1338.327593] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1338.356758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.422590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.131@tcp -r 1 -m GET -l 3 [ 1338.468603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.131@tcp -r 1 -m GET -e local_error [ 1338.512291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.131@tcp -r 1 -m PUT -l 6 [ 1339.891644] Lustre: DEBUG MARKER: Force oleg231-client.virtnet to discover 192.168.202.131@tcp (in background) [ 1340.042549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.131@tcp [ 1341.450823] Lustre: DEBUG MARKER: Force oleg231-server to discover 192.168.202.31@tcp [ 1343.137691] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1344.023790] Lustre: DEBUG MARKER: Wait for 68566 [ 1345.216964] Lustre: DEBUG MARKER: Finished wait on 68566 [ 1345.548086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1345.602874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1345.603137] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1345.805452] LNet: 69017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.811750] LNetError: 69017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.829478] LNet: Removed LNI 192.168.202.31@tcp [ 1346.182693] Key type .llcrypt unregistered [ 1346.184483] Key type ._llcrypt unregistered [ 1360.911187] Key type ._llcrypt registered [ 1360.912850] Key type .llcrypt registered [ 1362.473382] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:01:00 (1761238860) [ 1362.984332] Key type .llcrypt unregistered [ 1362.987361] Key type ._llcrypt unregistered [ 1367.986050] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1368.105454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1368.108191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1368.115739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1370.583712] Key type ._llcrypt registered [ 1370.589142] Key type .llcrypt registered [ 1370.733273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1370.749871] alg: No test for adler32 (adler32-zlib) [ 1371.658872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.701792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1371.729477] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1372.279355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1372.299673] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1379.855300] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:01:18 (1761238878) [ 1379.994578] LNet: 71090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.040614] LNetError: 71090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.043985] LNet: Removed LNI 192.168.202.31@tcp [ 1380.074161] LNet: Removed LNI 10.1.2.3@tcp [ 1380.731581] Key type .llcrypt unregistered [ 1380.733217] Key type ._llcrypt unregistered [ 1385.725986] Key type ._llcrypt registered [ 1385.727874] Key type .llcrypt registered [ 1385.887816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1385.901109] alg: No test for adler32 (adler32-zlib) [ 1386.796379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.845166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.875128] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1387.480069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1387.496205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1388.388309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.31@tcp [ 1389.473547] LNetError: Unexpected error -22 connecting to 192.168.202.31@tcp at host 192.168.202.31:988 [ 1393.734373] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:01:32 (1761238892) [ 1393.866164] LNet: 72214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.877537] LNetError: 72214:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.886141] LNet: Removed LNI 192.168.202.31@tcp [ 1393.904596] LNet: Removed LNI 10.1.2.3@tcp [ 1394.439457] Key type .llcrypt unregistered [ 1394.445710] Key type ._llcrypt unregistered [ 1399.173105] Key type ._llcrypt registered [ 1399.174803] Key type .llcrypt registered [ 1399.372286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.387652] alg: No test for adler32 (adler32-zlib) [ 1400.284861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.384463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.404980] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1401.055455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1401.078131] LNet: Added LNI 192.168.202.31@tcp2 [8/256/0/180] [ 1401.883523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.31@tcp1 --nid 192.168.202.31@tcp2 [ 1405.739288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp1 192.168.202.31@tcp2 [ 1405.763413] LNetError: 73043:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.31@tcp1 to 192.168.202.31@tcp2 - Route table may be misconfigured [ 1405.785428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp2 192.168.202.31@tcp1 [ 1411.171337] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:01:49 (1761238909) [ 1411.346341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.360559] LNet: 73615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.371186] LNetError: 73615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.385803] LNet: Removed LNI 192.168.202.31@tcp1 [ 1412.455990] LNet: Removed LNI 192.168.202.31@tcp2 [ 1412.496449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.593768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.634101] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1413.284038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.297545] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1414.152594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1414.201289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.31@tcp [ 1414.231384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp1 192.168.202.31@tcp1 [ 1414.271214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.31@tcp -d 192.168.202.31@tcp -e network_timeout [ 1414.304466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -e network_timeout [ 1414.333797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -e network_timeout [ 1414.362686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -e network_timeout [ 1414.399905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1414.416292] LNet: There was an unexpected network error while writing to 192.168.202.31: rc = -22 [ 1414.433202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1414.529085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1414.620685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1414.626055] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1419.868076] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:01:58 (1761238918) [ 1420.036882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.072404] LNet: 74473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.075674] LNetError: 74473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.082392] LNetError: 74473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1420.088989] LNet: Removed LNI 192.168.202.31@tcp [ 1421.163109] LNet: Removed LNI 192.168.202.31@tcp1 [ 1421.181498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.253787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1421.466329] LNet: 74509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.883126] Key type .llcrypt unregistered [ 1421.884861] Key type ._llcrypt unregistered [ 1432.209893] Key type ._llcrypt registered [ 1432.216320] Key type .llcrypt registered [ 1433.426652] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:02:11 (1761238931) [ 1433.888839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.907527] alg: No test for adler32 (adler32-zlib) [ 1434.780619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.076857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1437.548250] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:02:16 (1761238936) [ 1437.633302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.666055] LNet: 75598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.702726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.791830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1437.823282] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1438.528771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1438.570374] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1439.405906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1439.452317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp1 [ 1439.506873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp1 [ 1445.026964] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:02:23 (1761238943) [ 1446.692421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg231-server [ 1448.347650] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:02:26 (1761238946) [ 1450.579333] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg231-server [ 1452.493695] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:02:30 (1761238950) [ 1454.460365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg231-server [ 1455.907934] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:02:34 (1761238954) [ 1457.334591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg231-server [ 1458.856425] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:02:37 (1761238957) [ 1460.240649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg231-server [ 1461.589463] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:02:40 (1761238960) [ 1463.092505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg231-server [ 1464.633441] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:02:43 (1761238963) [ 1466.429370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg231-server [ 1467.903280] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:02:46 (1761238966) [ 1469.626754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg231-server [ 1471.265894] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:02:49 (1761238969) [ 1472.879455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg231-server [ 1474.833496] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:02:53 (1761238973) [ 1474.930713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.953050] LNet: 79469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.957944] LNetError: 79469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.999617] LNet: Removed LNI 192.168.202.31@tcp [ 1476.004969] LNetError: 79469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.018058] LNet: Removed LNI 192.168.202.31@tcp1 [ 1476.047644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.121644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.148563] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1476.888589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1476.953045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1477.199973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.248346] LNet: 79620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.260819] LNetError: 79620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.281185] LNet: Removed LNI 192.168.202.31@tcp [ 1477.327480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.403916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1477.446655] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1478.226468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1478.525096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.655184] LNet: Removed LNI 192.168.202.31@tcp [ 1479.709542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.787552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.821894] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1480.685840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1480.761353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1480.958323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.983148] LNet: 79906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.986592] LNet: 79906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1480.993055] LNetError: 79906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.997503] LNetError: 79906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.014758] LNet: Removed LNI 192.168.202.31@tcp [ 1481.038115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.145243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1481.208047] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1482.202575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1482.483166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.560935] LNet: Removed LNI 192.168.202.31@tcp [ 1483.599863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.717526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.776392] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1484.771800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1484.865992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1485.181574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.207414] LNet: 80193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.210759] LNet: 80193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1485.221234] LNetError: 80193:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.227633] LNetError: 80193:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1485.240242] LNet: Removed LNI 192.168.202.31@tcp [ 1485.259306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.324723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1485.369168] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1486.110454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1486.357827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.461822] LNet: Removed LNI 192.168.202.31@tcp [ 1487.505542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1487.572909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1487.594720] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1488.370467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1488.432567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1490.944665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.009557] LNet: Removed LNI 192.168.202.31@tcp [ 1492.045408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.109128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1492.152827] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1492.794603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1493.072409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.183356] LNet: Removed LNI 192.168.202.31@tcp [ 1494.202914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.249977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.289136] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1494.831806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1494.892380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1495.111759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.140396] LNet: 80784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.146278] LNet: 80784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1495.157792] LNetError: 80784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.161779] LNetError: 80784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1496.235606] LNet: Removed LNI 192.168.202.31@tcp [ 1496.273383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.357909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1496.385483] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1497.148328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1498.534897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.625811] LNet: Removed LNI 192.168.202.31@tcp [ 1499.666144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.750363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.799976] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1500.453766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1500.519344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1500.798415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.862696] LNet: Removed LNI 192.168.202.31@tcp [ 1501.904923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.979960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1502.022876] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1502.889193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1503.271837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.364458] LNet: Removed LNI 192.168.202.31@tcp [ 1504.385528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.461422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.497303] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1505.236458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1505.292223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.325184] LNet: Removed LNI 192.168.202.31@tcp [ 1505.358708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.411456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.445743] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1506.023366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1510.934280] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:03:29 (1761239009) [ 1511.124112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.144219] LNet: 82052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.147448] LNet: 82052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1511.164180] LNet: Removed LNI 192.168.202.31@tcp [ 1511.180914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.230076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1511.248933] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1512.091414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.109533] LNetError: 82187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.114088] LNetError: 82187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1512.134175] LNet: Removed LNI 192.168.202.31@tcp231 [ 1512.154379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.218949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1512.242820] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1512.249809] LNet: 82195:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1512.607762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1512.644236] LNet: Removed LNI 192.168.202.31@tcp231 [ 1512.658080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1512.690565] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1513.575802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.603455] LNet: Removed LNI 192.168.202.31@tcp231 [ 1513.618907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.668908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1513.688611] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/0] [ 1513.694855] LNet: 82345:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1514.040565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1514.071305] LNet: Removed LNI 192.168.202.31@tcp231 [ 1514.086109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1514.116026] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/0] [ 1514.881975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.911904] LNet: Removed LNI 192.168.202.31@tcp231 [ 1514.929380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.984799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1515.008234] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/60] [ 1515.014871] LNet: 82493:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1515.328383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1515.367450] LNet: Removed LNI 192.168.202.31@tcp231 [ 1515.382869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1515.403460] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/60] [ 1516.265747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.294543] LNet: Removed LNI 192.168.202.31@tcp231 [ 1516.315450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.374810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1516.416703] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1516.803896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1516.833775] LNet: Removed LNI 192.168.202.31@tcp231 [ 1516.858434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1516.920714] LNet: Added LNI 192.168.202.31@tcp231 [8/256/0/180] [ 1523.735422] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:03:42 (1761239022) [ 1524.050030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.112363] LNet: Removed LNI 192.168.202.31@tcp231 [ 1524.148055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.241992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1524.271925] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1524.854452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1524.879477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1524.897571] LNet: Removed LNI 192.168.202.31@tcp232 [ 1524.913245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1524.928476] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1525.494591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1525.533710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1525.572834] LNet: Removed LNI 192.168.202.31@tcp232 [ 1525.587768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1525.621343] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1526.172945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1526.208879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1526.241978] LNet: Removed LNI 192.168.202.31@tcp232 [ 1526.271466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1526.300808] LNet: Added LNI 192.168.202.31@tcp232 [8/256/0/180] [ 1526.888322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1526.935718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1526.974200] LNet: Removed LNI 192.168.202.31@tcp232 [ 1527.004217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1532.348881] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:03:51 (1761239031) [ 1533.618892] Key type .llcrypt unregistered [ 1533.625325] Key type ._llcrypt unregistered [ 1538.631502] Key type ._llcrypt registered [ 1538.634178] Key type .llcrypt registered [ 1538.788629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1538.799929] alg: No test for adler32 (adler32-zlib) [ 1539.693245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1539.731621] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1546.705683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1554.925772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.966322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.990338] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1556.527166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_interrupt -r 2 [ 1556.554948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 2 [ 1556.588812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_interrupt -r 2 [ 1556.632937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 2 [ 1556.673883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_interrupt -r 2 [ 1556.713186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_interrupt -r 2 [ 1556.756580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_interrupt -r 2 [ 1556.802326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_interrupt -r 2 [ 1556.831362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1556.858783] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1556.869217] LNet: 1 local NIs in recovery (showing 1): 192.168.202.31@tcp [ 1557.242658] LNet: 85111:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1557.319722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1557.380760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1557.890038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_dropped -r 2 [ 1557.923724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 2 [ 1557.952736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_dropped -r 2 [ 1558.004747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 2 [ 1558.045676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_dropped -r 2 [ 1558.079599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_dropped -r 2 [ 1558.108341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_dropped -r 2 [ 1558.132773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_dropped -r 2 [ 1558.164942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1558.186182] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1558.535501] LNet: 85219:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1558.646919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1558.700606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1559.291289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_aborted -r 2 [ 1559.365476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 2 [ 1559.397814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_aborted -r 2 [ 1559.425908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 2 [ 1559.462065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_aborted -r 2 [ 1559.484203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_aborted -r 2 [ 1559.531684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_aborted -r 2 [ 1559.563736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_aborted -r 2 [ 1559.601216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1559.616118] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1559.951491] LNet: 85328:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.051600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.094486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1560.598858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_no_route -r 2 [ 1560.631908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 2 [ 1560.668127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_no_route -r 2 [ 1560.706748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 2 [ 1560.739774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_no_route -r 2 [ 1560.787059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_no_route -r 2 [ 1560.830659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_no_route -r 2 [ 1560.888241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_no_route -r 2 [ 1560.959938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1561.513022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1561.580910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1562.097540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e local_timeout -r 2 [ 1562.144883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 2 [ 1562.208832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e local_timeout -r 2 [ 1562.253657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 2 [ 1562.294234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e local_timeout -r 2 [ 1562.331136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e local_timeout -r 2 [ 1562.353835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e local_timeout -r 2 [ 1562.394972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e local_timeout -r 2 [ 1562.463968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1562.495677] LNet: There was an unexpected network error while writing to 192.168.202.131: rc = -22 [ 1562.498680] LNet: Skipped 1 previous similar message [ 1562.831677] LNet: 85544:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1562.836114] LNet: 85544:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1562.889524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1562.937843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1563.416366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp -m GET -e remote_dropped -r 2 [ 1563.486957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 2 [ 1563.574043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp -m GET -e remote_dropped -r 2 [ 1563.638043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 2 [ 1563.719903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp -m GET -e remote_dropped -r 2 [ 1563.770362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.131@tcp1 -m GET -e remote_dropped -r 2 [ 1563.823529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp -m GET -e remote_dropped -r 2 [ 1563.853827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.31@tcp1 -d 192.168.202.31@tcp1 -m GET -e remote_dropped -r 2 [ 1563.919706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.31@tcp 192.168.202.131@tcp [ 1563.943977] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.131@tcp [ 1564.371826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1564.408752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1569.963491] LNet: 85721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.981275] LNetError: 85721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.039311] LNet: Removed LNI 192.168.202.31@tcp [ 1571.041337] LNetError: 85721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.070371] LNet: Removed LNI 192.168.202.31@tcp1 [ 1571.673783] Key type .llcrypt unregistered [ 1571.691688] Key type ._llcrypt unregistered [ 1590.038548] Key type ._llcrypt registered [ 1590.040124] Key type .llcrypt registered [ 1591.474097] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:04:49 (1761239089) [ 1591.985705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.003184] alg: No test for adler32 (adler32-zlib) [ 1593.012822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.082359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.104242] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1593.868874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1593.953137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1594.024352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1594.127047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1594.185105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1594.254132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1601.122495] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:04:59 (1761239099) [ 1601.193649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.245057] LNet: 87272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.263145] LNetError: 87272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.320444] LNet: Removed LNI 192.168.202.31@tcp [ 1601.355319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.422422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1601.463407] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1608.823675] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:05:07 (1761239107) [ 1609.079494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.105310] LNet: 87976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.116823] LNetError: 87976:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.132719] LNet: Removed LNI 192.168.202.31@tcp [ 1609.157949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.218693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.263286] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1616.459527] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:05:14 (1761239114) [ 1617.935882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1619.672284] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:05:18 (1761239118) [ 1620.115723] LNet: 89009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.122982] LNetError: 89009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.157256] LNet: Removed LNI 192.168.202.31@tcp [ 1620.725749] Key type .llcrypt unregistered [ 1620.735463] Key type ._llcrypt unregistered [ 1626.475569] Key type ._llcrypt registered [ 1626.477480] Key type .llcrypt registered [ 1626.677315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1626.699535] alg: No test for adler32 (adler32-zlib) [ 1627.584267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1627.610561] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1634.082649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1638.055341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.131@tcp [ 1638.410208] LNet: 89419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.416291] LNetError: 89419:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.438808] LNet: Removed LNI 192.168.202.31@tcp [ 1638.858486] Key type .llcrypt unregistered [ 1638.867102] Key type ._llcrypt unregistered [ 1648.625801] Key type ._llcrypt registered [ 1648.627643] Key type .llcrypt registered [ 1649.749183] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1651.151834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1652.976609] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:05:51 (1761239151) [ 1653.527630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.539693] alg: No test for adler32 (adler32-zlib) [ 1654.484489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.599119] LNet: 90502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.134410] Key type .llcrypt unregistered [ 1655.136080] Key type ._llcrypt unregistered [ 1660.817161] Key type ._llcrypt registered [ 1660.818820] Key type .llcrypt registered [ 1661.058654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1661.082560] alg: No test for adler32 (adler32-zlib) [ 1662.021717] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1662.024696] LNet: 90754:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1662.062194] LNetError: 90766:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.34@tcp is being used as a gateway but routing feature is not turned on [ 1662.067888] LNetError: 90766:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.34@tcp has gone from up to down [ 1662.830532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.865384] LNet: 90812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.879352] LNetError: 90812:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.911308] LNet: Removed LNI 192.168.202.31@tcp [ 1662.949876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.023221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1663.063151] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1663.071271] LNet: 90819:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1663.082367] LNet: 90819:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1663.088731] LNetError: 90819:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.32@tcp is a gateway. Can not delete it [ 1663.094332] LNetError: 90819:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.202.32@tcp is a router. Can not be deleted [ 1663.104779] LNet: 90819:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.119075] LNetError: 90816:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.34@tcp is being used as a gateway but routing feature is not turned on [ 1663.137568] LNetError: 90816:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 1663.144958] LNetError: 90816:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.34@tcp has gone from up to down [ 1663.157836] LNetError: 90816:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1669.655976] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:06:08 (1761239168) [ 1671.828727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg231-server [ 1673.338695] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:06:11 (1761239171) [ 1674.981958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg231-server [ 1676.704277] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:06:15 (1761239175) [ 1678.138037] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 1680.032734] Key type lgssc registered [ 1681.194657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1699.966414] Key type lgssc unregistered [ 1700.376620] LNet: 93255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.389644] LNetError: 93255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.416360] LNet: Removed LNI 192.168.202.31@tcp [ 1701.182337] Key type .llcrypt unregistered [ 1701.183769] Key type ._llcrypt unregistered [ 1714.535572] Key type ._llcrypt registered [ 1714.536784] Key type .llcrypt registered [ 1715.833394] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:06:54 (1761239214) [ 1716.223298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.249580] alg: No test for adler32 (adler32-zlib) [ 1717.145662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.222733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1717.257791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1717.295830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1717.332362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1723.659442] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:07:02 (1761239222) [ 1723.777887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.823949] LNet: 94650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.894552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.959420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.984789] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1724.869925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.885429] LNet: 94798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.888659] LNetError: 94798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.912381] LNet: Removed LNI 192.168.202.31@tcp [ 1724.930077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.980783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1725.020926] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1725.497666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.523828] LNetError: 94826:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.537449] LNet: Removed LNI 192.168.202.31@tcp [ 1725.572903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.671714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1725.725197] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1731.505134] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:07:10 (1761239230) [ 1731.742941] LNet: 95416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.747668] LNet: 95416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1731.752737] LNetError: 95416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.771919] LNet: Removed LNI 192.168.202.31@tcp [ 1732.275709] Key type .llcrypt unregistered [ 1732.277555] Key type ._llcrypt unregistered [ 1737.535302] Key type ._llcrypt registered [ 1737.536512] Key type .llcrypt registered [ 1737.685666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1737.699889] alg: No test for adler32 (adler32-zlib) [ 1738.563557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.631698] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1738.633543] LNet: *** cfs_fail_loc=e003, val=100*** [ 1739.189951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1739.207472] LNet: *** cfs_fail_loc=e003, val=100*** [ 1739.208851] LNet: Skipped 2 previous similar messages [ 1739.219449] Ping reply received with mapping: [ 1739.221108] found metadata reply NI[192.168.202.31@tcp] [ 1739.222382] 1f ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1739.225317] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1739.227597] 5a 5a 5a 5a 5a 5a 5a 5a [ 1739.335835] LNet: 95806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.353524] LNetError: 95806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.369792] LNet: Removed LNI 192.168.202.31@tcp [ 1739.865692] Key type .llcrypt unregistered [ 1739.868336] Key type ._llcrypt unregistered [ 1748.838813] Key type ._llcrypt registered [ 1748.840445] Key type .llcrypt registered [ 1749.898422] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:07:28 (1761239248) [ 1750.437574] Key type .llcrypt unregistered [ 1750.443850] Key type ._llcrypt unregistered [ 1755.695292] Key type ._llcrypt registered [ 1755.696980] Key type .llcrypt registered [ 1755.903945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1755.928838] alg: No test for adler32 (adler32-zlib) [ 1757.135211] LNetError: 96841:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1757.529221] Key type .llcrypt unregistered [ 1757.531658] Key type ._llcrypt unregistered [ 1762.646942] Key type ._llcrypt registered [ 1762.649224] Key type .llcrypt registered [ 1764.050972] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:07:42 (1761239262) [ 1764.501691] Key type .llcrypt unregistered [ 1764.507526] Key type ._llcrypt unregistered [ 1769.579624] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1769.581932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1772.192768] Key type ._llcrypt registered [ 1772.194340] Key type .llcrypt registered [ 1772.443914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1772.468463] alg: No test for adler32 (adler32-zlib) [ 1773.373159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.443488] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1782.882453] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:08:01 (1761239281) [ 1782.974945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.008527] LNet: 98603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.015685] LNetError: 98603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1783.030414] LNet: Removed LNI 10.1.2.3@tcp [ 1783.050109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.853002] Key type .llcrypt unregistered [ 1783.854502] Key type ._llcrypt unregistered [ 1788.550745] Key type ._llcrypt registered [ 1788.557059] Key type .llcrypt registered [ 1788.761243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1788.799589] alg: No test for adler32 (adler32-zlib) [ 1789.755387] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1789.819962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1789.959718] LNet: 98895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.971888] LNetError: 98895:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.021641] LNet: Removed LNI 192.168.202.31@tcp [ 1791.761663] Key type .llcrypt unregistered [ 1791.770371] Key type ._llcrypt unregistered [ 1802.801344] Key type ._llcrypt registered [ 1802.803032] Key type .llcrypt registered [ 1803.872618] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:08:22 (1761239302) [ 1804.656546] Key type .llcrypt unregistered [ 1804.658178] Key type ._llcrypt unregistered [ 1811.149655] Key type ._llcrypt registered [ 1811.151157] Key type .llcrypt registered [ 1811.358812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1811.381779] alg: No test for adler32 (adler32-zlib) [ 1818.981185] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:08:37 (1761239317) [ 1819.198839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.235896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.309764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.358811] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1820.269304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.326866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1820.387486] LNet: Removed LNI 192.168.202.31@tcp [ 1820.666642] LNet: 100627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.960957] Key type .llcrypt unregistered [ 1820.962698] Key type ._llcrypt unregistered [ 1831.060286] Key type ._llcrypt registered [ 1831.064673] Key type .llcrypt registered [ 1832.305252] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:08:50 (1761239330) [ 1832.946798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.974364] alg: No test for adler32 (adler32-zlib) [ 1833.882385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.923768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.944962] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1834.554957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1834.609942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.31@tcp [ 1834.625608] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1834.629693] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.633285] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 0 [ 1834.637106] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1834.640148] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1834.644089] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.646695] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 0 [ 1834.649698] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1834.652244] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1834.655921] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.659136] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 1884 [ 1834.663482] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1834.665979] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1834.668956] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1834.672628] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1834, 0, 1884 [ 1834.676691] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1840.318615] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:08:58 (1761239338) [ 1840.860357] LNet: 102163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.867896] LNetError: 102163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.880680] LNet: Removed LNI 192.168.202.31@tcp [ 1841.378484] Key type .llcrypt unregistered [ 1841.381163] Key type ._llcrypt unregistered [ 1846.517896] Key type ._llcrypt registered [ 1846.524220] Key type .llcrypt registered [ 1846.706987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1846.724600] alg: No test for adler32 (adler32-zlib) [ 1847.630863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1847.672203] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1854.352927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 1861.366720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.417719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1861.441661] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 1862.630070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1862.771661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1862.775383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1862.779412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1865.027081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1865.064253] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1873.626310] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:09:32 (1761239372) [ 1873.745175] LNet: 103651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.750068] LNetError: 103651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.783684] LNet: Removed LNI 192.168.202.31@tcp [ 1874.796609] LNetError: 103651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.800179] LNet: Removed LNI 192.168.202.31@tcp1 [ 1874.819042] LNet: Removed LNI 10.1.2.3@tcp99 [ 1875.339952] Key type .llcrypt unregistered [ 1875.342132] Key type ._llcrypt unregistered [ 1880.774644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1880.780512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1883.355935] Key type ._llcrypt registered [ 1883.365303] Key type .llcrypt registered [ 1883.658135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1883.680347] alg: No test for adler32 (adler32-zlib) [ 1884.686870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.763442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.792476] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1885.752220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1885.784564] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1888.454090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1888.543700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1888.627068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1888.648644] LNetError: 104391:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1888.690700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1888.744380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1889.109386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1889.152192] LNet: 104411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.159340] LNetError: 104411:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.162503] LNet: Removed LNI 192.168.202.31@tcp [ 1889.171694] LNet: Removed LNI 10.1.2.3@tcp [ 1889.800492] Key type .llcrypt unregistered [ 1889.804181] Key type ._llcrypt unregistered [ 1901.488844] Key type ._llcrypt registered [ 1901.497407] Key type .llcrypt registered [ 1902.757064] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:10:01 (1761239401) [ 1903.344300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.363804] alg: No test for adler32 (adler32-zlib) [ 1904.287322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.333888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.370761] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1904.967525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.31@tcp [ 1905.051831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg231-client@tcp [ 1910.592827] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:10:09 (1761239409) [ 1910.749344] LNet: 105931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.770116] LNetError: 105931:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.787087] LNet: Removed LNI 192.168.202.31@tcp [ 1911.345553] Key type .llcrypt unregistered [ 1911.347108] Key type ._llcrypt unregistered [ 1916.775740] Key type ._llcrypt registered [ 1916.777491] Key type .llcrypt registered [ 1916.991153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.011900] alg: No test for adler32 (adler32-zlib) [ 1917.947749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1917.998565] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1918.037358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1923.819360] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:10:22 (1761239422) [ 1925.325327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1927.037630] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:10:25 (1761239425) [ 1928.610419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1929.996987] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:10:28 (1761239428) [ 1930.127664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.143373] LNet: 107352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.147394] LNetError: 107352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.164659] LNet: Removed LNI 192.168.202.31@tcp [ 1930.180605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.221650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.252889] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1930.935910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1930.989362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1936.931090] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:10:35 (1761239435) [ 1937.176881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.211082] LNet: 108058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.215584] LNetError: 108058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.239347] LNet: Removed LNI 192.168.202.31@tcp [ 1937.261246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.313182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1937.352521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1937.937218] Key type .llcrypt unregistered [ 1937.943157] Key type ._llcrypt unregistered [ 1946.393480] Key type ._llcrypt registered [ 1946.394981] Key type .llcrypt registered [ 1947.420796] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:10:46 (1761239446) [ 1947.813230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1947.830775] alg: No test for adler32 (adler32-zlib) [ 1948.734500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.787707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.816494] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1949.318242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1949.380872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1954.597615] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:10:53 (1761239453) [ 1954.789789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1954.823154] LNet: 109606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.826560] LNetError: 109606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.842214] LNet: Removed LNI 192.168.202.31@tcp [ 1954.868791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.931925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1954.962572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1959.781560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1961.077880] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:10:59 (1761239459) [ 1961.286383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.309739] LNet: 110325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.334832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.551619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1961.576422] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1962.200096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1962.219602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1962.919745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.940531] LNet: 110592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.948060] LNetError: 110592:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.955772] LNet: Removed LNI 192.168.202.31@tcp [ 1962.971989] LNet: Removed LNI 10.1.2.3@tcp [ 1962.997433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.070805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1963.102843] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1963.111513] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1963.590955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1963.611761] LNet: Removed LNI 192.168.202.31@tcp [ 1963.625482] LNet: Removed LNI 10.1.2.3@tcp [ 1968.954817] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:11:07 (1761239467) [ 1969.224040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.246512] LNet: 111193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.268787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.369790] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1969.526850] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1969.533113] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1969.540953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1970.763726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.804722] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 1971.422581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1971.435555] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1973.274459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.31@tcp [ 1973.309125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1973.357989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.31@tcp -m PUT -e local_timeout -r 1 [ 1973.411477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1973.479788] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1973.484591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1973.648498] LNetError: 111243:0:(lib-move.c:4947:lnet_parse()) 192.168.202.31@tcp, src 192.168.202.31@tcp: Dropping PUT (error -108 looking up sender) [ 1974.691826] LNet: 111695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.700873] LNetError: 111695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.709211] LNetError: 111695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1974.713247] LNet: Removed LNI 192.168.202.31@tcp [ 1974.739732] LNet: Removed LNI 10.1.2.3@tcp [ 1975.049232] Key type .llcrypt unregistered [ 1975.055668] Key type ._llcrypt unregistered [ 1985.979608] Key type ._llcrypt registered [ 1985.981112] Key type .llcrypt registered [ 1987.204935] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:11:25 (1761239485) [ 1987.644925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.659921] alg: No test for adler32 (adler32-zlib) [ 1988.604702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.960530] LNet: 112546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.523911] Key type .llcrypt unregistered [ 1989.525600] Key type ._llcrypt unregistered [ 1994.962762] Key type ._llcrypt registered [ 1994.964948] Key type .llcrypt registered [ 1995.170756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1995.185280] alg: No test for adler32 (adler32-zlib) [ 1996.116707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1996.155966] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2002.896102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 2003.089267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2003.650978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.131@tcp [ 2004.011667] LNet: 112983:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2004.087432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2004.118616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2004.649762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2005.015274] LNet: 113088:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2005.084598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2005.127371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2005.788630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.131@tcp [ 2006.068427] LNet: 113193:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2006.132560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2006.157134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2006.640157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2007.029483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2007.061420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2007.312743] LNet: 113339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.323456] LNetError: 113339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.368894] LNet: Removed LNI 192.168.202.31@tcp [ 2008.738704] Key type .llcrypt unregistered [ 2008.747667] Key type ._llcrypt unregistered [ 2021.390282] Key type ._llcrypt registered [ 2021.391924] Key type .llcrypt registered [ 2022.413963] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:12:01 (1761239521) [ 2022.969830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2022.978987] alg: No test for adler32 (adler32-zlib) [ 2023.921638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.287893] LNet: 114223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.886472] Key type .llcrypt unregistered [ 2024.888128] Key type ._llcrypt unregistered [ 2029.641157] Key type ._llcrypt registered [ 2029.642714] Key type .llcrypt registered [ 2029.857923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2029.870589] alg: No test for adler32 (adler32-zlib) [ 2030.747964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2030.774116] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2037.346231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.131@tcp [ 2045.184711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.263430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2045.280409] LNet: Added LNI 192.168.202.31@tcp1 [8/256/0/180] [ 2046.375448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2047.101163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.131@tcp [ 2047.199386] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.131@tcp [ 2047.592091] LNet: 114859:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2047.723319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2047.785565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2048.561223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2048.844224] LNet: 114964:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2048.921164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2048.949847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2049.581763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.131@tcp [ 2049.888817] LNet: 115070:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2049.937647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2049.960483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2050.493583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2050.915690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2050.959926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2054.969124] LNet: 115250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.974945] LNetError: 115250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.980495] LNet: Removed LNI 192.168.202.31@tcp [ 2054.989821] LNet: Removed LNI 192.168.202.31@tcp1 [ 2055.292639] Key type .llcrypt unregistered [ 2055.294358] Key type ._llcrypt unregistered [ 2069.630394] Key type ._llcrypt registered [ 2069.632216] Key type .llcrypt registered [ 2070.855750] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1846 sec ========== 13:12:49 (1761239569) [ 2071.556659] Key type .llcrypt unregistered [ 2071.560450] Key type ._llcrypt unregistered [ 2082.648387] Key type ._llcrypt registered [ 2082.650639] Key type .llcrypt registered [ 2083.101835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2083.118600] alg: No test for adler32 (adler32-zlib) [ 2084.153221] Lustre: Lustre: Build Version: 2.16.59_38_g3dab24e [ 2084.366915] LNet: Added LNI 192.168.202.31@tcp [8/256/0/180] [ 2086.023175] Key type lgssc registered [ 2087.092521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2141.335324] Lustre: Mounted lustre-client [ 2146.339541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2158.375765] LustreError: 118470:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff987920ebd000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2158.393927] LustreError: 118470:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2158.434200] Lustre: Unmounted lustre-client [ 2199.579590] Key type lgssc unregistered [ 2199.978831] LNet: 118954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.986103] LNetError: 118954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.000239] LNet: Removed LNI 192.168.202.31@tcp [ 2201.932326] Key type .llcrypt unregistered [ 2201.939316] Key type ._llcrypt unregistered