[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 542869532 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003087] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007643] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013135] *** VALIDATE selinux *** [ 0.021731] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026322] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] *** VALIDATE tmpfs *** [ 0.030433] *** VALIDATE proc *** [ 0.031000] *** VALIDATE cgroup *** [ 0.031012] *** VALIDATE cgroup2 *** [ 0.032374] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035033] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039251] debug: unmapping init [mem 0xffffffff85459000-0xffffffff85460fff] [ 0.041166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043029] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.044716] ... generic registers: 4 [ 0.045007] ... value mask: 0000ffffffffffff [ 0.045925] ... max period: 00007fffffffffff [ 0.046011] ... fixed-purpose events: 3 [ 0.047007] ... event mask: 000000070000000f [ 0.049216] rcu: Hierarchical SRCU implementation. [ 0.051406] smp: Bringing up secondary CPUs ... [ 0.052478] x86: Booting SMP configuration: [ 0.053026] .... node #0, CPUs: #1 #2 #3 [ 0.057009] smp: Brought up 1 node, 4 CPUs [ 0.059018] smpboot: Max logical packages: 1 [ 0.060016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.197649] node 0 deferred pages initialised in 135ms [ 0.202262] devtmpfs: initialized [ 0.203396] x86/mm: Memory block size: 128MB [ 0.205000] gcov: version magic: 0x41383552 [ 0.207415] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.208081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.210330] pinctrl core: initialized pinctrl subsystem [ 0.212202] [ 0.212601] ************************************************************* [ 0.215014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.217012] ** ** [ 0.220011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.221010] ** ** [ 0.223011] ** This means that this kernel is built to expose internal ** [ 0.225014] ** IOMMU data structures, which may compromise security on ** [ 0.228014] ** your system. ** [ 0.230013] ** ** [ 0.232013] ** If you see this message and you are not debugging the ** [ 0.234014] ** kernel, report this immediately to your vendor! ** [ 0.237012] ** ** [ 0.239024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241015] ************************************************************* [ 0.245107] NET: Registered protocol family 16 [ 0.246000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.246000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.246000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.246071] cpuidle: using governor menu [ 0.250210] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.253491] PCI: Using configuration type 1 for base access [ 0.255134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265165] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.267023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.271164] cryptd: max_cpu_qlen set to 1000 [ 0.274414] ACPI: Added _OSI(Module Device) [ 0.276019] ACPI: Added _OSI(Processor Device) [ 0.277000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282232] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.288665] ACPI: Interpreter enabled [ 0.290092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.291015] ACPI: Using IOAPIC for interrupt routing [ 0.293150] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.296616] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.306984] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.308053] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.311023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.313107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.318476] acpiphp: Slot [2] registered [ 0.319170] acpiphp: Slot [3] registered [ 0.321123] acpiphp: Slot [4] registered [ 0.322116] acpiphp: Slot [5] registered [ 0.323140] acpiphp: Slot [6] registered [ 0.324105] acpiphp: Slot [7] registered [ 0.326100] acpiphp: Slot [8] registered [ 0.327144] acpiphp: Slot [9] registered [ 0.328118] acpiphp: Slot [10] registered [ 0.329109] acpiphp: Slot [11] registered [ 0.330115] acpiphp: Slot [12] registered [ 0.331130] acpiphp: Slot [13] registered [ 0.333128] acpiphp: Slot [14] registered [ 0.334167] acpiphp: Slot [15] registered [ 0.335149] acpiphp: Slot [16] registered [ 0.337137] acpiphp: Slot [17] registered [ 0.338110] acpiphp: Slot [18] registered [ 0.339128] acpiphp: Slot [19] registered [ 0.341158] acpiphp: Slot [20] registered [ 0.342135] acpiphp: Slot [21] registered [ 0.343131] acpiphp: Slot [22] registered [ 0.345123] acpiphp: Slot [23] registered [ 0.346117] acpiphp: Slot [24] registered [ 0.347127] acpiphp: Slot [25] registered [ 0.348152] acpiphp: Slot [26] registered [ 0.349121] acpiphp: Slot [27] registered [ 0.350147] acpiphp: Slot [28] registered [ 0.352115] acpiphp: Slot [29] registered [ 0.353129] acpiphp: Slot [30] registered [ 0.354117] acpiphp: Slot [31] registered [ 0.356072] PCI host bridge to bus 0000:00 [ 0.357022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.360031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.363038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.365036] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.367028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373603] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.376000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.384798] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.390064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.391019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.393024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.395017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.398501] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.400645] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.402033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.405550] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408888] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.417015] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.421015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.426734] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449015] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.459075] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.487018] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.501000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.504330] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.506332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.508365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.510209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.514036] iommu: Default domain type: Passthrough [ 0.515621] SCSI subsystem initialized [ 0.516125] ACPI: bus type USB registered [ 0.517151] usbcore: registered new interface driver usbfs [ 0.519168] usbcore: registered new interface driver hub [ 0.521123] usbcore: registered new device driver usb [ 0.523190] pps_core: LinuxPPS API ver. 1 registered [ 0.525011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.527170] PTP clock support registered [ 0.531052] EDAC MC: Ver: 3.0.0 [ 0.533037] PCI: Using ACPI for IRQ routing [ 0.536339] NetLabel: Initializing [ 0.538016] NetLabel: domain hash size = 128 [ 0.540015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.543182] NetLabel: unlabeled traffic allowed by default [ 0.547062] vgaarb: loaded [ 0.549212] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.552023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.558000] clocksource: Switched to clocksource kvm-clock [ 0.667311] VFS: Disk quotas dquot_6.6.0 [ 0.668860] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.672445] *** VALIDATE ramfs *** [ 0.673690] *** VALIDATE hugetlbfs *** [ 0.675866] pnp: PnP ACPI init [ 0.678297] pnp: PnP ACPI: found 6 devices [ 0.696285] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.699706] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.702735] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.705190] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.707842] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.710355] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.713232] NET: Registered protocol family 2 [ 0.715689] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.720549] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.724326] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.729877] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.733480] TCP: Hash tables configured (established 65536 bind 65536) [ 0.736290] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.739045] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.741609] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.744111] NET: Registered protocol family 1 [ 0.746217] RPC: Registered named UNIX socket transport module. [ 0.748783] RPC: Registered udp transport module. [ 0.750161] RPC: Registered tcp transport module. [ 0.751561] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.753565] NET: Registered protocol family 44 [ 0.755377] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.759179] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.761923] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.764324] PCI: CLS 0 bytes, default 64 [ 0.765870] Unpacking initramfs... [ 2.214385] debug: unmapping init [mem 0xffff9a4bbcc64000-0xffff9a4bbffcffff] [ 2.220139] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.222527] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.226290] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.756099] Initialise system trusted keyrings [ 2.757715] Key type blacklist registered [ 2.760955] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.769764] zbud: loaded [ 2.773459] *** VALIDATE nfs *** [ 2.774590] *** VALIDATE nfs4 *** [ 2.775863] pstore: using deflate compression [ 2.779372] Platform Keyring initialized [ 3.036842] NET: Registered protocol family 38 [ 3.038836] Key type asymmetric registered [ 3.040813] Asymmetric key parser 'x509' registered [ 3.043369] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.047183] io scheduler mq-deadline registered [ 3.048972] io scheduler kyber registered [ 3.055847] io scheduler bfq registered [ 3.058981] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.062732] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.065861] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.069387] ACPI: Power Button [PWRF] [ 3.182531] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.342839] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.476473] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.504322] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.539437] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.546606] Non-volatile memory driver v1.3 [ 3.547996] Linux agpgart interface v0.103 [ 3.579433] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.582995] vda: detected capacity change from 0 to 68222976 [ 3.609179] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.612566] vdb: detected capacity change from 0 to 1073741824 [ 3.627948] libphy: Fixed MDIO Bus: probed [ 3.638053] usbcore: registered new interface driver usbserial_generic [ 3.640993] usbserial: USB Serial support registered for generic [ 3.643717] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.647974] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.650080] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.652585] mousedev: PS/2 mouse device common for all mice [ 3.655469] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.659469] rtc_cmos 00:05: RTC can wake from S4 [ 3.664124] rtc_cmos 00:05: registered as rtc0 [ 3.667666] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.673967] intel_pstate: CPU model not supported [ 3.676834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.682062] hid: raw HID events driver (C) Jiri Kosina [ 3.683922] usbcore: registered new interface driver usbhid [ 3.688804] usbhid: USB HID core driver [ 3.691162] drop_monitor: Initializing network drop monitor service [ 3.695859] Initializing XFRM netlink socket [ 3.699483] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.700036] NET: Registered protocol family 10 [ 3.708722] Segment Routing with IPv6 [ 3.710715] NET: Registered protocol family 17 [ 3.713347] mpls_gso: MPLS GSO support [ 3.719483] RAS: Correctable Errors collector initialized. [ 3.721470] AVX version of gcm_enc/dec engaged. [ 3.723124] AES CTR mode by8 optimization enabled [ 3.862501] sched_clock: Marking stable (3862466994, 0)->(4766468600, -904001606) [ 3.874073] registered taskstats version 1 [ 3.876625] Loading compiled-in X.509 certificates [ 3.880188] zswap: loaded using pool lzo/zbud [ 3.922938] Key type big_key registered [ 3.941579] Key type encrypted registered [ 3.943479] ima: No TPM chip found, activating TPM-bypass! [ 3.945591] ima: Allocated hash algorithm: sha1 [ 3.947433] ima: No architecture policies found [ 3.949075] evm: Initialising EVM extended attributes: [ 3.951220] evm: security.selinux [ 3.952506] evm: security.ima [ 3.953602] evm: security.capability [ 3.955124] evm: HMAC attrs: 0x1 [ 3.959716] rtc_cmos 00:05: setting system clock to 2025-10-10 19:42:24 UTC (1760125344) [ 3.965720] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 3.968760] debug: unmapping init [mem 0xffffffff85182000-0xffffffff85458fff] [ 3.977811] Write protecting the kernel read-only data: 28672k [ 3.984135] debug: unmapping init [mem 0xffffffff83803000-0xffffffff839fffff] [ 3.987592] debug: unmapping init [mem 0xffffffff84114000-0xffffffff841fffff] [ 4.030864] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.038789] systemd[1]: Detected virtualization kvm. [ 4.042270] systemd[1]: Detected architecture x86-64. [ 4.044234] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.070265] systemd[1]: No hostname configured. [ 4.072502] systemd[1]: Set hostname to . [ 4.074370] random: systemd: uninitialized urandom read (16 bytes read) [ 4.076476] systemd[1]: Initializing machine ID from random generator. [ 4.143853] random: ln: uninitialized urandom read (6 bytes read) [ 4.377884] random: systemd: uninitialized urandom read (16 bytes read) [ 4.380858] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.389892] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.398878] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.552244] device-mapper: uevent: version 1.0.3 [ 5.555791] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 6.840679] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.009562] random: fast init done [ 7.388190] scsi host0: ata_piix [ 7.433646] scsi host1: ata_piix [ 7.435025] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.436944] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 11.470500] random: crng init done [ 11.471628] random: 7 urandom warning(s) missed due to ratelimiting [ 13.694770] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 15.551975] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.025106] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.155809] hrtimer: interrupt took 9335011 ns [ 19.330917] SELinux: Disabled at runtime. [ 19.423588] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.451502] systemd[1]: Detected virtualization kvm. [ 19.455215] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.501079] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.507677] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.516450] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.539074] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.551761] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.575199] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.652445] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ 22.835559] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.220488] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.662255] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.861048] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.246531] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.366227] EDAC sbridge: Ver: 1.1.2 [ 29.227979] Key type dns_resolver registered [ 29.777340] NFS: Registering the id_resolver key type [ 29.789932] Key type id_resolver registered [ 29.791555] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg256-client login: [ 85.412756] libcfs: loading out-of-tree module taints kernel. [ 85.461639] Key type ._llcrypt registered [ 85.463128] Key type .llcrypt registered [ 85.817527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 85.830276] alg: No test for adler32 (adler32-zlib) [ 87.114720] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 87.717299] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 89.399176] Key type lgssc registered [ 90.268242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 180.237944] Lustre: Mounted lustre-client [ 184.110860] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.910658] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing check_logdir /tmp/testlogs/ [ 201.878149] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing yml_node [ 205.792216] Lustre: lustre-OST0000-osc-ffff9a4c033dd800: disconnect after 24s idle [ 206.223961] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 208.661130] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 210.725400] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 211.908045] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Oct 10 15:45:51 EDT 2025 [ 224.670049] LustreError: 7917:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9a4c033dd800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 224.682314] LustreError: 7917:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 224.705529] Lustre: Unmounted lustre-client [ 258.262972] Key type lgssc unregistered [ 258.543298] LNet: 8365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.552143] LNetError: 8365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.626783] LNet: Removed LNI 192.168.202.56@tcp [ 260.262150] Key type .llcrypt unregistered [ 260.263914] Key type ._llcrypt unregistered [ 279.506752] Key type ._llcrypt registered [ 279.514049] Key type .llcrypt registered [ 279.695427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 279.713520] alg: No test for adler32 (adler32-zlib) [ 280.575272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 280.612330] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 280.649848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 283.363801] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 283.505954] LNet: 9238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.510394] LNetError: 9238:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 283.527531] LNet: Removed LNI 192.168.202.56@tcp [ 284.060439] Key type .llcrypt unregistered [ 284.062076] Key type ._llcrypt unregistered [ 288.006615] Key type ._llcrypt registered [ 288.008513] Key type .llcrypt registered [ 289.231994] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:47:08 (1760125628) [ 289.448309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 289.457178] alg: No test for adler32 (adler32-zlib) [ 290.348160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.397291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.409201] LNet: 9669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.425889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.469519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 290.483170] LNet: 9676:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 295.344245] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:47:14 (1760125634) [ 295.397505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.408599] LNet: 10246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.420712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.478132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 300.567279] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:47:19 (1760125639) [ 300.772334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.793030] LNet: 10837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.817494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.879719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 305.737427] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:47:25 (1760125645) [ 305.967857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.991533] LNet: 11421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.007962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.081514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 310.838697] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:47:30 (1760125650) [ 310.932708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.951838] LNet: 12002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.969534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.032504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 311.306723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 316.817668] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:47:36 (1760125656) [ 316.959558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.020611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.078539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 317.428810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 317.741265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 322.660715] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:47:42 (1760125662) [ 322.721852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.736048] LNet: 13185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.740495] LNet: 13185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 322.753482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.824481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 327.201166] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:47:46 (1760125666) [ 327.244989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.267560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.316184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 327.339167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 327.554096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 327.576660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 327.799232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 327.823169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 328.077814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 328.124658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 328.428568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 328.458664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 328.704722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 328.723723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 328.972556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 329.000489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 329.270367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 329.303659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 329.605755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 329.636764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 329.856529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 329.888344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 330.141974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 330.166679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 330.450536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 330.491508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 330.797369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 330.831625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 335.135412] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:47:54 (1760125674) [ 335.277411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.300661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.360269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 335.396963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 339.587478] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:47:59 (1760125679) [ 339.775590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.790786] LNet: 15044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.796046] LNet: 15044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 339.819127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.872916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 339.902637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 343.912376] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:48:03 (1760125683) [ 344.033178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.081654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.143950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 344.165226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 349.015418] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:48:08 (1760125688) [ 349.165106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.208781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.252776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 349.276580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 354.156436] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:48:13 (1760125693) [ 354.394681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.429571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.492097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 354.530509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 358.991307] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:48:18 (1760125698) [ 359.160546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.205490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.273607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 359.308266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 363.868512] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:48:23 (1760125703) [ 363.959553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.007774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.097732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 364.508600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 369.237298] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:48:28 (1760125708) [ 369.337736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.387218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.563144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 369.763685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 374.260210] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:48:33 (1760125713) [ 374.334898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.356167] LNet: 19184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.359752] LNet: 19184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 374.385962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.505370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 374.664533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 379.384346] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:48:38 (1760125718) [ 379.518139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.552630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.860056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 380.018542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 383.910579] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:48:43 (1760125723) [ 383.970069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.003149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.054776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 384.080830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 388.028072] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:48:47 (1760125727) [ 388.134170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.174301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.219132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.267962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 392.579737] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:48:52 (1760125732) [ 392.645171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.668494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.726616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 396.947862] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:48:56 (1760125736) [ 397.052538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.092617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.159789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 401.541858] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:49:01 (1760125741) [ 401.597861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.623809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.676459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 401.948176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 406.525872] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:49:06 (1760125746) [ 406.610698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.638748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.694995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 406.717441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 410.991719] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:49:10 (1760125750) [ 411.099545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.126990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.189087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 411.218452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 414.960714] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:49:14 (1760125754) [ 415.056164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.089214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.145522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 415.166261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 419.034328] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:49:18 (1760125758) [ 419.136463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.163516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.217280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 419.243526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 423.003279] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:49:22 (1760125762) [ 423.052793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.085982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.124917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 423.170463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 423.182699] LNetError: 25771:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 423.202339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 423.220481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 426.541493] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:49:26 (1760125766) [ 426.648551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.675890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.727045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HOltNRljjX [ 426.771773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HOltNRljjX [ 426.809032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.HOltNRljjX [ 430.164879] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:49:29 (1760125769) [ 430.231882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.261617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.289648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 430.308226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 430.331978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 433.687836] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:49:33 (1760125773) [ 433.774674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.815593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.861136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 435.150286] LNet: Message forwarding will be enabled in 2s [ 439.263152] LNet: Message forwarding enabled [ 439.482586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 439.495553] LNet: Message forwarding will stop in 2s to 6s [ 443.802833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 444.289960] LNet: Message forwarding disabled [ 445.340470] LNet: Message forwarding will be enabled in 2s [ 448.415771] LNet: Message forwarding enabled [ 449.727742] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:49:49 (1760125789) [ 449.803717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.819970] LNet: 28221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.825536] LNet: 28221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 450.570537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.619105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 454.303851] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:49:53 (1760125793) [ 454.371616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.400781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.441214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 454.466195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 454.485266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 454.508268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 454.534039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 454.560026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 454.580559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 454.609236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 454.637579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 454.659518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 454.682790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 454.702942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 454.731572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 454.750263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 454.770321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 454.788636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 454.813023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 454.831884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 454.849698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 454.864515] LNetError: 28847:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 458.479384] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:49:58 (1760125798) [ 458.528727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.572355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.657590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.635658] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:50:02 (1760125802) [ 463.632503] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg256-server [ 464.614542] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:50:04 (1760125804) [ 464.717446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.737883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.764649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 464.779244] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 465.696912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[57-61]@tcp [ 465.709675] LNet: 30559:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 465.740954] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.58@tcp is being used as a gateway but routing feature is not turned on [ 465.755519] LNetError: 30352:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.58@tcp has gone from up to down [ 466.727784] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 466.734411] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 467.749193] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.57@tcp is being used as a gateway but routing feature is not turned on [ 467.755971] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 468.768507] LNetError: 30358:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.59@tcp has gone from up to down [ 468.779877] LNetError: 30358:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 469.112138] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:50:08 (1760125808) [ 469.797321] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.57@tcp is being used as a gateway but routing feature is not turned on [ 469.804482] LNetError: 30352:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 470.015380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg256-server [ 470.919718] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:50:10 (1760125810) [ 470.985808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.839193] LNetError: 30351:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.202.57@tcp [ 472.032729] LNetError: 31491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.049290] LNet: Removed LNI 192.168.202.56@tcp [ 472.061429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.100855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 472.123304] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 472.561592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[57-62/2]@tcp [ 472.576818] LNet: 31607:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 472.585854] LNet: 31607:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 472.611477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[57-62/2]@tcp [ 472.612099] LNetError: 31497:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.57@tcp has gone from up to down [ 472.634654] LNetError: 31497:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 476.069720] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:50:15 (1760125815) [ 476.142761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.156180] LNetError: 32195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 476.171202] LNet: Removed LNI 192.168.202.56@tcp [ 476.185419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.242862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 476.276129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.304182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.342289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.380919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.402604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.437188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 476.484928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.517384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.564349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.612074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.640159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.677387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 476.721298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.746803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.792713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.841822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.867347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.908840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 476.959629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.993090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.045171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.096953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.119471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.151343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 477.196799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.221343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.275273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.332761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.356244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.395601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 477.429541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.463589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.501975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 481.436950] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:50:21 (1760125821) [ 482.563835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg256-server [ 483.574874] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:50:23 (1760125823) [ 484.708634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg256-server [ 485.939413] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:50:25 (1760125825) [ 486.430525] Key type .llcrypt unregistered [ 486.433376] Key type ._llcrypt unregistered [ 489.380478] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 489.447514] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 489.451452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 489.454502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 491.725808] Key type ._llcrypt registered [ 491.727312] Key type .llcrypt registered [ 491.815432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 491.825235] alg: No test for adler32 (adler32-zlib) [ 492.658855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.691346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 492.707283] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 493.053353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 493.067033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 494.157176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 494.169729] LNet: Removed LNI 10.1.2.3@tcp [ 494.249784] LNet: 34308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.255601] LNetError: 34308:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 494.266599] LNet: Removed LNI 192.168.202.56@tcp [ 494.603113] Key type .llcrypt unregistered [ 494.604819] Key type ._llcrypt unregistered [ 500.317391] Key type ._llcrypt registered [ 500.318854] Key type .llcrypt registered [ 501.067837] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:50:40 (1760125840) [ 501.362711] Key type .llcrypt unregistered [ 501.364566] Key type ._llcrypt unregistered [ 504.261257] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 504.312924] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 504.315601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 504.318230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 506.635426] Key type ._llcrypt registered [ 506.638339] Key type .llcrypt registered [ 506.748525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 506.757514] alg: No test for adler32 (adler32-zlib) [ 507.609788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.644691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 507.664210] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 508.123798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 508.140703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 509.696537] LNet: 35786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.701381] LNetError: 35786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 509.704882] LNet: Removed LNI 192.168.202.56@tcp [ 509.718429] LNet: Removed LNI 10.1.2.3@tcp [ 510.143462] Key type .llcrypt unregistered [ 510.145481] Key type ._llcrypt unregistered [ 515.560028] Key type ._llcrypt registered [ 515.561573] Key type .llcrypt registered [ 516.360910] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:50:56 (1760125856) [ 516.661489] Key type .llcrypt unregistered [ 516.662946] Key type ._llcrypt unregistered [ 519.737522] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 519.822499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 519.825386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 519.828053] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 522.192923] Key type ._llcrypt registered [ 522.194483] Key type .llcrypt registered [ 522.314174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 522.329264] alg: No test for adler32 (adler32-zlib) [ 523.183400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.229295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 523.249147] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 524.425632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 524.437232] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 525.507449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 525.517659] LNet: Removed LNI 10.1.2.3@tcp [ 525.527635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 525.548118] LNet: Removed LNI 10.1.2.31@tcp [ 525.558141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 525.571823] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 525.952395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 525.968115] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 526.459665] LNet: 37686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.463663] LNetError: 37686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 526.469706] LNet: Removed LNI 10.1.2.3@tcp [ 526.477328] LNet: Removed LNI 10.1.2.31@tcp [ 526.775308] Key type .llcrypt unregistered [ 526.776981] Key type ._llcrypt unregistered [ 532.098808] Key type ._llcrypt registered [ 532.100454] Key type .llcrypt registered [ 532.860225] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:51:12 (1760125872) [ 533.132689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 533.143832] alg: No test for adler32 (adler32-zlib) [ 534.032838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.075507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 534.091493] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 534.562516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.579852] LNet: 38630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.585928] LNetError: 38630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.600868] LNet: Removed LNI 192.168.202.56@tcp [ 534.613419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.655304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.56@tcp [ 534.677750] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 535.135806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.56@tcp [ 535.171283] LNet: Removed LNI 192.168.202.56@tcp [ 535.216394] LNet: 38762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.672975] Key type .llcrypt unregistered [ 535.678647] Key type ._llcrypt unregistered [ 541.436204] Key type ._llcrypt registered [ 541.437994] Key type .llcrypt registered [ 542.148437] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:51:21 (1760125881) [ 543.115380] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg256-server [ 543.951322] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:51:23 (1760125883) [ 544.175496] Key type .llcrypt unregistered [ 544.176878] Key type ._llcrypt unregistered [ 546.935395] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 547.004822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 547.006636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 547.008844] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 549.303351] Key type ._llcrypt registered [ 549.304744] Key type .llcrypt registered [ 549.409978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 549.419984] alg: No test for adler32 (adler32-zlib) [ 550.272762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.305494] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 550.754258] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 552.129914] LNet: 40585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.133430] LNetError: 40585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.136891] LNet: Removed LNI 192.168.202.56@tcp1 [ 552.147250] LNet: Removed LNI 10.1.2.3@tcp2 [ 552.572244] Key type .llcrypt unregistered [ 552.574069] Key type ._llcrypt unregistered [ 558.524627] Key type ._llcrypt registered [ 558.527139] Key type .llcrypt registered [ 559.318692] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:51:38 (1760125898) [ 559.547532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.559137] alg: No test for adler32 (adler32-zlib) [ 560.397592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.427740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 560.442561] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 560.991605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.015104] LNet: 41541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.019409] LNetError: 41541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 561.032132] LNet: Removed LNI 192.168.202.56@tcp [ 561.055262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.126250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 561.142218] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 565.038729] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:51:44 (1760125904) [ 565.118154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.134733] LNet: 42132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.141279] LNetError: 42132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 565.150687] LNet: Removed LNI 192.168.202.56@tcp [ 565.163616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.198394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 565.215226] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 565.717754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.56 [ 565.734025] LNet: Removed LNI 192.168.202.56@tcp [ 569.156016] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:51:48 (1760125908) [ 569.241627] LNet: 42834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.245813] LNetError: 42834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.258026] LNet: Removed LNI 192.168.202.56@tcp [ 569.607743] Key type .llcrypt unregistered [ 569.609787] Key type ._llcrypt unregistered [ 572.904422] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 573.075381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 573.077734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 573.082346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 574.467809] Key type ._llcrypt registered [ 574.469342] Key type .llcrypt registered [ 574.576121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 574.585245] alg: No test for adler32 (adler32-zlib) [ 575.684811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 579.416629] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:51:59 (1760125919) [ 579.509332] LNet: 43724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.515913] LNetError: 43724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 579.534285] LNet: Removed LNI 10.1.2.3@tcp [ 579.906522] Key type .llcrypt unregistered [ 579.908215] Key type ._llcrypt unregistered [ 582.930780] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 583.087686] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 583.090200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 583.092986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 585.533754] Key type ._llcrypt registered [ 585.535553] Key type .llcrypt registered [ 585.634472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 585.643668] alg: No test for adler32 (adler32-zlib) [ 586.550961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 590.073056] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:52:09 (1760125929) [ 590.132786] LNet: 44607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.141861] LNetError: 44607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 590.153861] LNet: Removed LNI 10.1.2.3@tcp [ 590.531335] Key type .llcrypt unregistered [ 590.535196] Key type ._llcrypt unregistered [ 594.253329] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 594.448214] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 594.457334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 594.463384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 596.948439] Key type ._llcrypt registered [ 596.951218] Key type .llcrypt registered [ 597.063062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 597.075568] alg: No test for adler32 (adler32-zlib) [ 597.987717] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 602.426118] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:52:21 (1760125941) [ 602.511318] LNet: 45500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.516931] LNetError: 45500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 602.532107] LNet: Removed LNI 10.1.2.3@tcp [ 602.954631] Key type .llcrypt unregistered [ 602.956789] Key type ._llcrypt unregistered [ 606.429521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 606.619458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 606.622793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 606.625419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 609.037286] Key type ._llcrypt registered [ 609.038721] Key type .llcrypt registered [ 609.134907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 609.151548] alg: No test for adler32 (adler32-zlib) [ 610.007283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.048330] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.710266] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:52:34 (1760125954) [ 614.963187] LNet: 46724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.970932] LNetError: 46724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 614.981767] LNet: Removed LNI 10.1.2.3@tcp [ 615.325821] Key type .llcrypt unregistered [ 615.327846] Key type ._llcrypt unregistered [ 618.078744] Key type ._llcrypt registered [ 618.080449] Key type .llcrypt registered [ 618.164134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.173876] alg: No test for adler32 (adler32-zlib) [ 619.023540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 619.040757] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 623.179652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 623.552410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_interrupt -r 1 [ 623.577530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_interrupt -r 1 [ 623.599738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 623.616192] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 623.791518] LNet: 47149:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 623.847962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 623.876134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 623.899495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 624.267238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_dropped -r 1 [ 624.290922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_dropped -r 1 [ 624.311939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 624.324920] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 624.524150] LNet: 47246:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 624.577646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 624.615523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 624.637215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 624.968022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_aborted -r 1 [ 624.993969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_aborted -r 1 [ 625.020910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 625.244418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 625.271267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 625.291191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 625.616267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_no_route -r 1 [ 625.656996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_no_route -r 1 [ 625.677243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 625.693468] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 625.697134] LNet: Skipped 1 previous similar message [ 625.887956] LNet: 47440:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 625.892586] LNet: 47440:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 625.938773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 625.958543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 625.980212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 626.341153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_timeout -r 1 [ 626.365511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_timeout -r 1 [ 626.396383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 626.690265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 626.726453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 626.751178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 627.110843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_error -r 1 [ 627.130831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_error -r 1 [ 627.151955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 627.373670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 627.395798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 627.414640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 627.575873] LNet: 47671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.581989] LNetError: 47671:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.592916] LNet: Removed LNI 192.168.202.56@tcp [ 627.854670] Key type .llcrypt unregistered [ 627.856599] Key type ._llcrypt unregistered [ 637.636097] Key type ._llcrypt registered [ 637.637970] Key type .llcrypt registered [ 638.564945] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:52:58 (1760125978) [ 639.165307] Key type .llcrypt unregistered [ 639.166978] Key type ._llcrypt unregistered [ 642.819478] Key type ._llcrypt registered [ 642.820838] Key type .llcrypt registered [ 643.003252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.019306] alg: No test for adler32 (adler32-zlib) [ 643.880745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 643.904295] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 648.548329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 655.156845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.196448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 655.208653] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 656.070138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_interrupt -r 1 [ 656.096822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_interrupt -r 1 [ 656.125552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_interrupt -r 1 [ 656.148973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_interrupt -r 1 [ 656.169926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_interrupt -r 1 [ 656.216903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_interrupt -r 1 [ 656.240771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_interrupt -r 1 [ 656.264224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_interrupt -r 1 [ 656.288345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 656.301115] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 656.304860] LNet: 1 local NIs in recovery (showing 1): 192.168.202.56@tcp [ 656.506087] LNet: 49131:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 656.558291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 656.582966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 656.608306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 656.934642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_dropped -r 1 [ 656.961383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_dropped -r 1 [ 656.994592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_dropped -r 1 [ 657.028586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_dropped -r 1 [ 657.056676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_dropped -r 1 [ 657.078915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_dropped -r 1 [ 657.103484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_dropped -r 1 [ 657.130410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_dropped -r 1 [ 657.163330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 657.180378] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 657.183503] LNet: Skipped 2 previous similar messages [ 657.391452] LNet: 49241:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 657.434791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 657.458894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 657.484288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 657.803435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_aborted -r 1 [ 657.825196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_aborted -r 1 [ 657.852250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_aborted -r 1 [ 657.874839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_aborted -r 1 [ 657.894941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_aborted -r 1 [ 657.920983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_aborted -r 1 [ 657.941420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_aborted -r 1 [ 657.964233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_aborted -r 1 [ 657.988236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 658.211280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 658.235959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 658.275560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 658.554172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_no_route -r 1 [ 658.577850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_no_route -r 1 [ 658.611427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_no_route -r 1 [ 658.634096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_no_route -r 1 [ 658.659706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_no_route -r 1 [ 658.683494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_no_route -r 1 [ 658.709270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_no_route -r 1 [ 658.734604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_no_route -r 1 [ 658.763858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 658.780783] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 658.784795] LNet: Skipped 7 previous similar messages [ 659.003455] LNet: 49459:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 659.006233] LNet: 49459:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 659.041122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 659.061889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 659.096498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 659.375890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_timeout -r 1 [ 659.408536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_timeout -r 1 [ 659.431710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_timeout -r 1 [ 659.461696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_timeout -r 1 [ 659.486639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_timeout -r 1 [ 659.510961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_timeout -r 1 [ 659.531451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_timeout -r 1 [ 659.555377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_timeout -r 1 [ 659.580341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 659.796842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 659.814340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 659.837258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 660.125884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_error -r 1 [ 660.153141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_error -r 1 [ 660.172473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_error -r 1 [ 660.194556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_error -r 1 [ 660.219878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_error -r 1 [ 660.246450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_error -r 1 [ 660.269493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_error -r 1 [ 660.300676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_error -r 1 [ 660.331933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 660.566995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 660.584772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 660.600921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 663.396930] LNet: 49748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.401654] LNetError: 49748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.408588] LNet: Removed LNI 192.168.202.56@tcp [ 664.425967] LNet: Removed LNI 192.168.202.56@tcp1 [ 664.694639] Key type .llcrypt unregistered [ 664.696084] Key type ._llcrypt unregistered [ 673.745032] Key type ._llcrypt registered [ 673.746938] Key type .llcrypt registered [ 674.656354] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:53:34 (1760126014) [ 675.157388] Key type .llcrypt unregistered [ 675.159258] Key type ._llcrypt unregistered [ 678.611688] Key type ._llcrypt registered [ 678.613135] Key type .llcrypt registered [ 678.804896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 678.819790] alg: No test for adler32 (adler32-zlib) [ 679.672596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 679.694181] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 684.654477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 685.100843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_dropped -r 1 [ 685.136143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_dropped -r 1 [ 685.167913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 685.194881] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 685.430300] LNet: 50982:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 685.492768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 685.517656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 685.544977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 685.934608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_error -r 1 [ 685.966887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_error -r 1 [ 686.010300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 686.037954] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 686.245919] LNet: 51079:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 686.300297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.332937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 686.357436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 686.726150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_timeout -r 1 [ 686.745589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_timeout -r 1 [ 686.774431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 687.059646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 687.078748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 687.104745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 687.279401] LNet: 51212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.288817] LNetError: 51212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.307388] LNet: Removed LNI 192.168.202.56@tcp [ 687.609809] Key type .llcrypt unregistered [ 687.611567] Key type ._llcrypt unregistered [ 696.586357] Key type ._llcrypt registered [ 696.587978] Key type .llcrypt registered [ 697.426956] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:53:57 (1760126037) [ 697.979662] Key type .llcrypt unregistered [ 697.981440] Key type ._llcrypt unregistered [ 701.380351] Key type ._llcrypt registered [ 701.382060] Key type .llcrypt registered [ 701.534531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 701.542325] alg: No test for adler32 (adler32-zlib) [ 702.404285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 702.429873] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 707.066206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 711.012129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.030971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 711.045735] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 711.699655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_dropped -r 1 [ 711.718524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e remote_dropped -r 1 [ 711.739918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_dropped -r 1 [ 711.766405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e remote_dropped -r 1 [ 711.793215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e remote_dropped -r 1 [ 711.813903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e remote_dropped -r 1 [ 711.838631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e remote_dropped -r 1 [ 711.862085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e remote_dropped -r 1 [ 711.885450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 711.898395] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 711.902071] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.156@tcp [ 712.083242] LNet: 52661:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 712.121868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 712.139541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 712.165742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 712.189723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 712.457805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_error -r 1 [ 712.482874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e remote_error -r 1 [ 712.503739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_error -r 1 [ 712.526071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e remote_error -r 1 [ 712.549547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e remote_error -r 1 [ 712.573447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e remote_error -r 1 [ 712.597777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e remote_error -r 1 [ 712.622633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e remote_error -r 1 [ 712.644457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 712.656296] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 712.659855] LNet: Skipped 2 previous similar messages [ 712.856532] LNet: 52772:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 712.898240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 712.919110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 712.944754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 712.966098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 713.273872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_timeout -r 1 [ 713.297752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e remote_timeout -r 1 [ 713.321441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_timeout -r 1 [ 713.348556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e remote_timeout -r 1 [ 713.373744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e remote_timeout -r 1 [ 713.404644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e remote_timeout -r 1 [ 713.428159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e remote_timeout -r 1 [ 713.455876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e remote_timeout -r 1 [ 713.475040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.156@tcp [ 713.698205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 713.728497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 713.753928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 713.775261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 716.843381] LNet: 52957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.852219] LNetError: 52957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.859572] LNet: Removed LNI 192.168.202.56@tcp [ 716.867611] LNet: Removed LNI 192.168.202.56@tcp1 [ 717.189849] Key type .llcrypt unregistered [ 717.192829] Key type ._llcrypt unregistered [ 725.410157] Key type ._llcrypt registered [ 725.411657] Key type .llcrypt registered [ 726.104638] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:54:25 (1760126065) [ 726.400366] Key type .llcrypt unregistered [ 726.401603] Key type ._llcrypt unregistered [ 729.117979] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 729.233292] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 729.235961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 729.238531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 731.523284] Key type ._llcrypt registered [ 731.524922] Key type .llcrypt registered [ 731.599813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.613691] alg: No test for adler32 (adler32-zlib) [ 732.453867] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 732.474564] LNet: 54041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.478809] LNetError: 54041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.492432] LNet: Removed LNI 192.168.202.56@tcp [ 732.897576] Key type .llcrypt unregistered [ 732.899209] Key type ._llcrypt unregistered [ 735.711281] Key type ._llcrypt registered [ 735.713625] Key type .llcrypt registered [ 735.811196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.823346] alg: No test for adler32 (adler32-zlib) [ 736.666030] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 736.668682] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 736.689483] LNet: 54310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.694990] LNetError: 54310:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.698675] LNet: Removed LNI 192.168.202.56@tcp [ 736.706881] LNet: Removed LNI 10.1.2.3@tcp1 [ 737.090072] Key type .llcrypt unregistered [ 737.093232] Key type ._llcrypt unregistered [ 740.066040] Key type ._llcrypt registered [ 740.068127] Key type .llcrypt registered [ 740.179624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.191412] alg: No test for adler32 (adler32-zlib) [ 741.050129] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 741.052427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.075704] LNet: 54579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.080156] LNetError: 54579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.084961] LNet: Removed LNI 192.168.202.56@tcp [ 741.091243] LNet: Removed LNI 10.1.2.3@tcp [ 741.492611] Key type .llcrypt unregistered [ 741.494354] Key type ._llcrypt unregistered [ 745.322041] Key type ._llcrypt registered [ 745.323775] Key type .llcrypt registered [ 745.461601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.470893] alg: No test for adler32 (adler32-zlib) [ 746.356177] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 746.358477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.383301] LNet: 54849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.388926] LNetError: 54849:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.392680] LNet: Removed LNI 192.168.202.56@tcp [ 746.402817] LNet: Removed LNI 10.1.2.3@tcp [ 746.913488] Key type .llcrypt unregistered [ 746.915206] Key type ._llcrypt unregistered [ 750.624584] Key type ._llcrypt registered [ 750.626283] Key type .llcrypt registered [ 750.758753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.772985] alg: No test for adler32 (adler32-zlib) [ 751.660584] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 751.662537] LNetError: 55109:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 751.666098] LNetError: Error -100 starting up LNI tcp [ 751.675534] LNet: Removed LNI 192.168.202.56@tcp [ 752.169326] Key type .llcrypt unregistered [ 752.171108] Key type ._llcrypt unregistered [ 758.975316] Key type ._llcrypt registered [ 758.976665] Key type .llcrypt registered [ 759.848361] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:54:59 (1760126099) [ 760.353690] Key type .llcrypt unregistered [ 760.355362] Key type ._llcrypt unregistered [ 763.780107] Key type ._llcrypt registered [ 763.781559] Key type .llcrypt registered [ 763.898860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.909671] alg: No test for adler32 (adler32-zlib) [ 764.764213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 764.786526] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 769.111705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 769.495771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e network_timeout -r 1 [ 769.529613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e network_timeout -r 1 [ 769.550976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 769.564271] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 769.727756] LNet: 56348:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 769.789057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 769.814176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 769.962392] LNet: 56382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.966446] LNetError: 56382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.980731] LNet: Removed LNI 192.168.202.56@tcp [ 770.218719] Key type .llcrypt unregistered [ 770.220698] Key type ._llcrypt unregistered [ 779.850348] Key type ._llcrypt registered [ 779.851936] Key type .llcrypt registered [ 780.014627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.028447] alg: No test for adler32 (adler32-zlib) [ 780.886600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 780.906440] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 785.728288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 790.912194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.935762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 790.950562] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 791.926296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e network_timeout -r 1 [ 791.957742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e network_timeout -r 1 [ 791.997150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e network_timeout -r 1 [ 792.034177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e network_timeout -r 1 [ 792.068640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e network_timeout -r 1 [ 792.101169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e network_timeout -r 1 [ 792.127687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e network_timeout -r 1 [ 792.160839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e network_timeout -r 1 [ 792.182964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 792.195773] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 792.376461] LNet: 57243:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 792.417523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 792.439505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 795.170629] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 795.231115] LNet: 1 local NIs in recovery (showing 1): 192.168.202.56@tcp [ 795.330645] LNet: 57311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.337930] LNetError: 57311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.342127] LNet: Removed LNI 192.168.202.56@tcp [ 795.351051] LNet: Removed LNI 192.168.202.56@tcp1 [ 795.631204] Key type .llcrypt unregistered [ 795.633172] Key type ._llcrypt unregistered [ 804.026294] Key type ._llcrypt registered [ 804.027917] Key type .llcrypt registered [ 804.767954] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:55:44 (1760126144) [ 804.995633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.008229] alg: No test for adler32 (adler32-zlib) [ 805.827564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.853464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.867207] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 806.227456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 806.239809] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 806.673698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 806.727155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 806.754468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 806.777442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 806.798925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 806.820549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 806.839200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.56@tcp [ 806.850261] LNetError: 58399:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.202.56@tcp to recovery [ 806.879437] LNet: 1 local NIs in recovery (showing 1): 192.168.202.56@tcp [ 806.879932] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 808.929504] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 809.051215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 812.002471] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 812.282632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 812.321390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 812.359506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.373206] LNet: 58497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.376790] LNetError: 58497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.379911] LNet: Removed LNI 192.168.202.56@tcp [ 812.388210] LNet: Removed LNI 192.168.202.56@tcp1 [ 812.402942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.434559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.449945] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 812.833679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 812.848425] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 813.369995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 813.398168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 813.434423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 813.465359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 813.488181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 813.511749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 813.531870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 813.559632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.56@tcp [ 813.575534] LNetError: 58763:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.202.56@tcp to recovery [ 814.496449] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 816.876407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 820.641834] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 820.645017] LNet: Skipped 1 previous similar message [ 825.418295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 825.437324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 825.457592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 828.797538] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:56:08 (1760126168) [ 828.837499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.854395] LNet: 59483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.861586] LNetError: 59483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.865185] LNetError: 59483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 828.868983] LNet: Removed LNI 192.168.202.56@tcp [ 828.877798] LNet: Removed LNI 192.168.202.56@tcp1 [ 828.900170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.934524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.950236] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 829.361955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 829.377251] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 829.832843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 829.870878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 829.890381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 829.909323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 829.934165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 829.959184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 829.980604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.56@tcp --health 0 [ 830.038567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 830.943647] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.56@tcp [ 830.948444] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 830.951266] LNet: Skipped 1 previous similar message [ 844.775893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 844.857247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 844.873308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 844.888155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.56@tcp --health 500 [ 844.976445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 851.288224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 851.322749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.336692] LNet: 59975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.340829] LNetError: 59975:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.344991] LNetError: 59975:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.349455] LNet: Removed LNI 192.168.202.56@tcp [ 851.362508] LNet: Removed LNI 192.168.202.56@tcp1 [ 851.380112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.425857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.441614] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 851.864416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 851.876954] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 852.271877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 852.290920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 852.318639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 852.338836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 852.361718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 852.380619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 852.398129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 852.416086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.56@tcp --health 0 [ 853.473899] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 853.476653] LNet: Skipped 3 previous similar messages [ 853.562315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 864.088675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 864.105448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 864.123795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 866.995552] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:56:46 (1760126206) [ 867.204218] LNet: 61025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.208818] LNetError: 61025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.213125] LNetError: 61025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 867.216519] LNet: Removed LNI 192.168.202.56@tcp [ 867.226129] LNet: Removed LNI 192.168.202.56@tcp1 [ 867.518153] Key type .llcrypt unregistered [ 867.519668] Key type ._llcrypt unregistered [ 869.932942] Key type ._llcrypt registered [ 869.935075] Key type .llcrypt registered [ 870.014858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.023479] alg: No test for adler32 (adler32-zlib) [ 870.842339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 870.860912] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 874.742129] Lustre: DEBUG MARKER: Initial discovery [ 874.771224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.156@tcp [ 875.795142] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 875.819190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 875.842408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.156@tcp [ 875.855717] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 875.866880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 875.887519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.156@tcp -r 1 -m GET -l 3 [ 875.907393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.156@tcp -r 1 -m GET -e local_error [ 875.927447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.156@tcp -r 1 -m PUT -l 6 [ 876.550577] Lustre: DEBUG MARKER: Force oleg256-client.virtnet to discover 192.168.202.156@tcp (in background) [ 876.589975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.156@tcp [ 877.177250] Lustre: DEBUG MARKER: Force oleg256-server to discover 192.168.202.56@tcp [ 878.165259] Lustre: DEBUG MARKER: Wait for 61774 [ 879.647237] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 880.225871] Lustre: DEBUG MARKER: Finished wait on 61774 [ 880.276605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 880.298089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 880.402909] LNet: 62230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.406260] LNetError: 62230:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.417294] LNet: Removed LNI 192.168.202.56@tcp [ 880.594493] Key type .llcrypt unregistered [ 880.596249] Key type ._llcrypt unregistered [ 887.452585] Key type ._llcrypt registered [ 887.454214] Key type .llcrypt registered [ 888.017487] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:57:07 (1760126227) [ 888.255338] Key type .llcrypt unregistered [ 888.256984] Key type ._llcrypt unregistered [ 890.507424] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.569051] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.572604] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.575044] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 892.811708] Key type ._llcrypt registered [ 892.813535] Key type .llcrypt registered [ 892.892949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 892.900667] alg: No test for adler32 (adler32-zlib) [ 893.724883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.754977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.768967] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 894.085976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 894.097865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.900536] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:57:17 (1760126237) [ 898.005285] LNet: 64290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.009064] LNetError: 64290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.014692] LNet: Removed LNI 192.168.202.56@tcp [ 898.022964] LNet: Removed LNI 10.1.2.3@tcp [ 898.365828] Key type .llcrypt unregistered [ 898.367053] Key type ._llcrypt unregistered [ 901.218092] Key type ._llcrypt registered [ 901.219882] Key type .llcrypt registered [ 901.337820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.346854] alg: No test for adler32 (adler32-zlib) [ 902.192879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.229819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.249445] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 902.672450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 902.690931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.242845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.56@tcp [ 904.288737] LNetError: Unexpected error -22 connecting to 192.168.202.56@tcp at host 192.168.202.56:988 [ 906.506509] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:57:26 (1760126246) [ 906.614872] LNet: 65410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.620852] LNetError: 65410:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.626376] LNet: Removed LNI 192.168.202.56@tcp [ 906.636630] LNet: Removed LNI 10.1.2.3@tcp [ 906.940472] Key type .llcrypt unregistered [ 906.942293] Key type ._llcrypt unregistered [ 909.893261] Key type ._llcrypt registered [ 909.895058] Key type .llcrypt registered [ 910.020576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.033565] alg: No test for adler32 (adler32-zlib) [ 910.848051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.875530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 910.889672] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 911.214370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 911.227086] LNet: Added LNI 192.168.202.56@tcp2 [8/256/0/180] [ 911.737077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.56@tcp1 --nid 192.168.202.56@tcp2 [ 914.020510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp1 192.168.202.56@tcp2 [ 914.031444] LNetError: 66237:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.56@tcp1 to 192.168.202.56@tcp2 - Route table may be misconfigured [ 914.045822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp2 192.168.202.56@tcp1 [ 917.286926] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:57:37 (1760126257) [ 917.346690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.362369] LNet: 66804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.367118] LNetError: 66804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.370318] LNet: Removed LNI 192.168.202.56@tcp1 [ 918.438732] LNet: Removed LNI 192.168.202.56@tcp2 [ 918.461379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.498623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 918.519104] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 918.913680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 918.929232] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 919.349620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 919.380492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.56@tcp [ 919.406455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp1 192.168.202.56@tcp1 [ 919.439289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.56@tcp -d 192.168.202.56@tcp -e network_timeout [ 919.464421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -e network_timeout [ 919.494947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -e network_timeout [ 919.523730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -e network_timeout [ 919.547469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 919.559047] LNet: There was an unexpected network error while writing to 192.168.202.56: rc = -22 [ 919.571213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 919.640117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 919.706072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 920.544330] LNet: 1 local NIs in recovery (showing 1): 192.168.202.56@tcp [ 922.819033] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:57:42 (1760126262) [ 922.918397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.934897] LNet: 67660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.940255] LNetError: 67660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.944297] LNetError: 67660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 922.949343] LNet: Removed LNI 192.168.202.56@tcp [ 924.006562] LNet: Removed LNI 192.168.202.56@tcp1 [ 924.020202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.089508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 924.306559] LNet: 67700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.568525] Key type .llcrypt unregistered [ 924.573620] Key type ._llcrypt unregistered [ 930.594628] Key type ._llcrypt registered [ 930.596318] Key type .llcrypt registered [ 931.326954] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:57:51 (1760126271) [ 931.578128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.588960] alg: No test for adler32 (adler32-zlib) [ 932.430374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.257912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 934.216599] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:57:54 (1760126274) [ 934.270205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.289706] LNet: 68789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.315828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.372864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.386263] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 934.764875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 934.781148] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 935.269161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 935.290619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp1 [ 935.311591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp1 [ 938.157208] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:57:58 (1760126278) [ 939.016802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg256-server [ 939.835753] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:57:59 (1760126279) [ 940.865033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg256-server [ 941.774819] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:58:01 (1760126281) [ 942.608855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg256-server [ 943.418788] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:58:03 (1760126283) [ 944.316600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg256-server [ 945.119877] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:58:04 (1760126284) [ 946.022343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg256-server [ 946.938703] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:58:06 (1760126286) [ 947.757797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg256-server [ 948.540708] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:58:08 (1760126288) [ 949.478978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg256-server [ 950.348715] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:58:10 (1760126290) [ 951.306804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg256-server [ 952.208528] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:58:12 (1760126292) [ 953.141223] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg256-server [ 953.935710] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:58:13 (1760126293) [ 954.003378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.017439] LNet: 72659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.022572] LNetError: 72659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.039789] LNet: Removed LNI 192.168.202.56@tcp [ 955.041767] LNetError: 72659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.052097] LNet: Removed LNI 192.168.202.56@tcp1 [ 955.065399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.094404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.109055] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 955.454708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 955.503115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 956.632102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.644989] LNet: 72813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.648114] LNetError: 72813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.661334] LNet: Removed LNI 192.168.202.56@tcp [ 956.674325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.704156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 956.720552] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 957.081216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 957.178914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.245661] LNet: Removed LNI 192.168.202.56@tcp [ 958.256684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.290691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.306147] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 958.690722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 958.746219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 958.872028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.891756] LNet: 73100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.895254] LNet: 73100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 958.899515] LNetError: 73100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.902834] LNetError: 73100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 958.914979] LNet: Removed LNI 192.168.202.56@tcp [ 958.931691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.963517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 958.979736] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 959.404941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 959.539184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.613857] LNet: Removed LNI 192.168.202.56@tcp [ 960.625224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.661836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.677968] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 961.021561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 961.056414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 961.191731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.277735] LNet: Removed LNI 192.168.202.56@tcp [ 962.289188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.322049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 962.341113] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 962.743367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 962.874800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.941280] LNet: Removed LNI 192.168.202.56@tcp [ 963.953151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.997211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.011030] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 964.401495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 964.458089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 965.679862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.695977] LNet: 73683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.698645] LNet: 73683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 965.703820] LNetError: 73683:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.706580] LNetError: 73683:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 965.722677] LNet: Removed LNI 192.168.202.56@tcp [ 965.733669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.767502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 965.790126] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 966.353090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 966.513536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.590514] LNet: Removed LNI 192.168.202.56@tcp [ 967.608202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.653529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.677634] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 968.134405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 968.185802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 969.417602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.475418] LNet: Removed LNI 192.168.202.56@tcp [ 970.497279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.546869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 970.573331] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 971.194818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 972.415644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.481801] LNet: Removed LNI 192.168.202.56@tcp [ 973.500293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.547868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.572900] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 974.069681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 974.122309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 974.284674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.304855] LNet: 74274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.308531] LNet: 74274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 974.311713] LNetError: 74274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.315642] LNetError: 74274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 974.330608] LNet: Removed LNI 192.168.202.56@tcp [ 974.347219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.394896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 974.410192] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 974.873370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 975.079296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.167529] LNet: Removed LNI 192.168.202.56@tcp [ 976.184641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.227127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 976.242277] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 976.656835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 976.690673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.715085] LNet: Removed LNI 192.168.202.56@tcp [ 976.727034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.760088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 976.786239] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 977.166981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 980.799156] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:58:40 (1760126320) [ 980.885523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.913769] LNet: Removed LNI 192.168.202.56@tcp [ 980.933188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.971810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 980.990782] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/180] [ 981.637330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.657413] LNet: Removed LNI 192.168.202.56@tcp231 [ 981.668130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.704468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 981.721316] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/180] [ 981.725327] LNet: 75392:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.918656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 981.935887] LNet: Removed LNI 192.168.202.56@tcp231 [ 981.946963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 981.965783] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/180] [ 982.513546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.532366] LNet: Removed LNI 192.168.202.56@tcp231 [ 982.547327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.582636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 982.599074] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/0] [ 982.603689] LNet: 75541:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.800626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 982.824183] LNet: Removed LNI 192.168.202.56@tcp231 [ 982.831973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 982.848836] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/0] [ 983.401131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.424777] LNet: Removed LNI 192.168.202.56@tcp231 [ 983.435521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.470923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 983.485931] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/60] [ 983.671857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 983.689085] LNet: Removed LNI 192.168.202.56@tcp231 [ 983.697310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 983.708806] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/60] [ 984.226938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.247487] LNet: Removed LNI 192.168.202.56@tcp231 [ 984.256723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.298337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 984.318053] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/180] [ 984.321414] LNet: 75837:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 984.325097] LNet: 75837:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 984.507591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 984.534652] LNet: Removed LNI 192.168.202.56@tcp231 [ 984.545346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 984.561772] LNet: Added LNI 192.168.202.56@tcp231 [8/256/0/180] [ 988.077787] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:58:47 (1760126327) [ 988.136987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.167872] LNet: Removed LNI 192.168.202.56@tcp231 [ 988.184422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.227865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 988.245874] LNet: Added LNI 192.168.202.56@tcp232 [8/256/0/180] [ 988.644055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 988.663759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 988.685235] LNet: Removed LNI 192.168.202.56@tcp232 [ 988.700065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 988.721122] LNet: Added LNI 192.168.202.56@tcp232 [8/256/0/180] [ 989.132420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 989.158148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 989.181504] LNet: Removed LNI 192.168.202.56@tcp232 [ 989.193280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 989.210291] LNet: Added LNI 192.168.202.56@tcp232 [8/256/0/180] [ 989.608148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 989.633284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 989.657662] LNet: Removed LNI 192.168.202.56@tcp232 [ 989.671432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 989.694804] LNet: Added LNI 192.168.202.56@tcp232 [8/256/0/180] [ 990.123172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 990.156395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 990.179129] LNet: Removed LNI 192.168.202.56@tcp232 [ 990.187759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 993.714052] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:58:53 (1760126333) [ 993.973568] LNet: 77659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.976653] LNet: 77659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 994.323319] Key type .llcrypt unregistered [ 994.325638] Key type ._llcrypt unregistered [ 997.763235] Key type ._llcrypt registered [ 997.764901] Key type .llcrypt registered [ 997.909259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.921068] alg: No test for adler32 (adler32-zlib) [ 998.788932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 998.823667] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1003.521717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 1007.277317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.297476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1007.314745] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 1007.951239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_interrupt -r 2 [ 1007.971115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_interrupt -r 2 [ 1007.993052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_interrupt -r 2 [ 1008.020884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_interrupt -r 2 [ 1008.039434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_interrupt -r 2 [ 1008.062817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_interrupt -r 2 [ 1008.082561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_interrupt -r 2 [ 1008.101837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_interrupt -r 2 [ 1008.122497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.156@tcp [ 1008.133575] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 1008.140898] LNet: 1 local NIs in recovery (showing 1): 192.168.202.56@tcp [ 1008.362196] LNet: 78305:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1008.410457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.435381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.705688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_dropped -r 2 [ 1008.729775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_dropped -r 2 [ 1008.750733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_dropped -r 2 [ 1008.776197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_dropped -r 2 [ 1008.798162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_dropped -r 2 [ 1008.839963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_dropped -r 2 [ 1008.869526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_dropped -r 2 [ 1008.897849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_dropped -r 2 [ 1008.926494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.156@tcp [ 1008.944993] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 1009.172863] LNet: 78413:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1009.223444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1009.260300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1009.538580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_aborted -r 2 [ 1009.560579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_aborted -r 2 [ 1009.583634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_aborted -r 2 [ 1009.605723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_aborted -r 2 [ 1009.626767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_aborted -r 2 [ 1009.646534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_aborted -r 2 [ 1009.670805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_aborted -r 2 [ 1009.700701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_aborted -r 2 [ 1009.732618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.156@tcp [ 1009.955632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1009.973331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1010.302044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_no_route -r 2 [ 1010.333984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_no_route -r 2 [ 1010.366911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_no_route -r 2 [ 1010.414607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_no_route -r 2 [ 1010.438180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_no_route -r 2 [ 1010.461582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_no_route -r 2 [ 1010.483375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_no_route -r 2 [ 1010.506086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_no_route -r 2 [ 1010.527377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.156@tcp [ 1010.541964] LNet: There was an unexpected network error while writing to 192.168.202.156: rc = -22 [ 1010.546906] LNet: Skipped 2 previous similar messages [ 1010.758549] LNet: 78630:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1010.762181] LNet: 78630:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1010.809389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1010.828933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.090872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e local_timeout -r 2 [ 1011.111705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e local_timeout -r 2 [ 1011.135565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e local_timeout -r 2 [ 1011.154771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e local_timeout -r 2 [ 1011.178415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e local_timeout -r 2 [ 1011.202994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e local_timeout -r 2 [ 1011.223091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e local_timeout -r 2 [ 1011.249903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e local_timeout -r 2 [ 1011.277265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.156@tcp [ 1011.514185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.532242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.817829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp -m GET -e remote_dropped -r 2 [ 1011.854965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.156@tcp1 -m GET -e remote_dropped -r 2 [ 1011.880607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp -m GET -e remote_dropped -r 2 [ 1011.914029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp -d 192.168.202.56@tcp1 -m GET -e remote_dropped -r 2 [ 1011.952671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp -m GET -e remote_dropped -r 2 [ 1011.976672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.156@tcp1 -m GET -e remote_dropped -r 2 [ 1012.009642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp -m GET -e remote_dropped -r 2 [ 1012.038357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.56@tcp1 -d 192.168.202.56@tcp1 -m GET -e remote_dropped -r 2 [ 1012.063114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.56@tcp 192.168.202.156@tcp [ 1012.080970] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.156@tcp [ 1012.316943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.340319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.120719] LNet: 78914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.123900] LNetError: 78914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.129214] LNet: Removed LNI 192.168.202.56@tcp [ 1015.135752] LNet: Removed LNI 192.168.202.56@tcp1 [ 1015.348471] Key type .llcrypt unregistered [ 1015.350163] Key type ._llcrypt unregistered [ 1022.598602] Key type ._llcrypt registered [ 1022.600267] Key type .llcrypt registered [ 1023.483896] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:59:23 (1760126363) [ 1023.679306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.690879] alg: No test for adler32 (adler32-zlib) [ 1024.539280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.567482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1024.584844] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1024.993924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1025.036505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1025.061715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1025.102202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1025.131361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1025.162078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1028.815512] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:59:28 (1760126368) [ 1028.895972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.910571] LNet: 80467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.917162] LNetError: 80467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.926819] LNet: Removed LNI 192.168.202.56@tcp [ 1028.938304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.980424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.001501] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1032.700103] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:59:32 (1760126372) [ 1032.799132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1032.811076] LNet: 81168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.815043] LNetError: 81168:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.825293] LNet: Removed LNI 192.168.202.56@tcp [ 1032.838380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.867439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1032.882178] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1036.425801] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:59:36 (1760126376) [ 1037.270654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1038.350740] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:59:38 (1760126378) [ 1038.604850] LNet: 82199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.608678] LNetError: 82199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.620825] LNet: Removed LNI 192.168.202.56@tcp [ 1038.990859] Key type .llcrypt unregistered [ 1038.992672] Key type ._llcrypt unregistered [ 1041.851426] Key type ._llcrypt registered [ 1041.853193] Key type .llcrypt registered [ 1041.976741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.984049] alg: No test for adler32 (adler32-zlib) [ 1042.819397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1042.841069] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1046.419700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 1048.518518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.156@tcp [ 1048.693301] LNet: 82609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.700303] LNetError: 82609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.716456] LNet: Removed LNI 192.168.202.56@tcp [ 1048.931491] Key type .llcrypt unregistered [ 1048.933085] Key type ._llcrypt unregistered [ 1053.726430] Key type ._llcrypt registered [ 1053.728183] Key type .llcrypt registered [ 1054.344749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1055.154874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1055.918162] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:59:55 (1760126395) [ 1056.074221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.080981] alg: No test for adler32 (adler32-zlib) [ 1056.915096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.961408] LNet: 83693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.263306] Key type .llcrypt unregistered [ 1057.265128] Key type ._llcrypt unregistered [ 1060.173664] Key type ._llcrypt registered [ 1060.175443] Key type .llcrypt registered [ 1060.266317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.276110] alg: No test for adler32 (adler32-zlib) [ 1061.101128] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1061.103197] LNet: 83944:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1061.128184] LNetError: 83956:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.58@tcp is being used as a gateway but routing feature is not turned on [ 1061.136555] LNetError: 83956:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.58@tcp has gone from up to down [ 1061.417422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.479225] LNetError: 83956:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.61@tcp has gone from up to down [ 1064.482660] LNetError: 83956:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 1064.486760] LNet: 84002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.491702] LNetError: 84002:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.505775] LNet: Removed LNI 192.168.202.56@tcp [ 1064.518316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.550506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1064.565227] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1064.570035] LNet: 84010:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1064.574696] LNet: 84010:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1064.581236] LNetError: 84010:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.58@tcp is a gateway. Can not delete it [ 1064.590703] LNetError: 84010:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.202.58@tcp is a router. Can not be deleted [ 1064.598515] LNet: 84010:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.605922] LNetError: 84007:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 1064.618049] LNetError: 84007:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 1067.295562] LNetError: 84012:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.61@tcp has gone from up to down [ 1067.309507] LNetError: 84012:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1068.165992] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:00:07 (1760126407) [ 1069.281348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg256-server [ 1070.354536] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:00:09 (1760126409) [ 1071.667777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg256-server [ 1073.145866] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:00:12 (1760126412) [ 1075.272843] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1077.240750] Key type lgssc registered [ 1079.113490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1104.558196] Key type lgssc unregistered [ 1105.048170] LNet: 86427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.061620] LNetError: 86427:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.091683] LNet: Removed LNI 192.168.202.56@tcp [ 1106.944036] Key type .llcrypt unregistered [ 1106.945430] Key type ._llcrypt unregistered [ 1124.983908] Key type ._llcrypt registered [ 1124.985434] Key type .llcrypt registered [ 1126.528554] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:01:05 (1760126465) [ 1126.937538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1126.966056] alg: No test for adler32 (adler32-zlib) [ 1127.940460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.014935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1128.060484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1128.114615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1128.152109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1134.526365] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:01:13 (1760126473) [ 1134.705767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1134.755104] LNet: 87841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.835805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.916792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1134.949891] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1135.765207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1135.785958] LNet: 87984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.791714] LNetError: 87984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.804211] LNet: Removed LNI 192.168.202.56@tcp [ 1135.823403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.890493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1135.915671] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1136.352237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1136.372772] LNetError: 88012:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.391181] LNet: Removed LNI 192.168.202.56@tcp [ 1136.409773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.458230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1136.476640] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1141.876990] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:01:21 (1760126481) [ 1141.978915] LNet: 88597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.989886] LNet: 88597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1142.003086] LNetError: 88597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.034129] LNet: Removed LNI 192.168.202.56@tcp [ 1142.530262] Key type .llcrypt unregistered [ 1142.533295] Key type ._llcrypt unregistered [ 1146.440326] Key type ._llcrypt registered [ 1146.444418] Key type .llcrypt registered [ 1146.629075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.641478] alg: No test for adler32 (adler32-zlib) [ 1147.516322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1147.574545] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1147.576749] LNet: *** cfs_fail_loc=e003, val=100*** [ 1148.110375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 1148.129772] LNet: *** cfs_fail_loc=e003, val=100*** [ 1148.131484] LNet: Skipped 2 previous similar messages [ 1148.138166] Ping reply received with mapping: [ 1148.140746] found metadata reply NI[192.168.202.56@tcp] [ 1148.151052] 38 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1148.153141] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1148.155216] 5a 5a 5a 5a 5a 5a 5a 5a [ 1148.240735] LNet: 88991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.250447] LNetError: 88991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.266307] LNet: Removed LNI 192.168.202.56@tcp [ 1148.720619] Key type .llcrypt unregistered [ 1148.723758] Key type ._llcrypt unregistered [ 1155.739566] Key type ._llcrypt registered [ 1155.740928] Key type .llcrypt registered [ 1156.744351] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:01:36 (1760126496) [ 1157.037740] Key type .llcrypt unregistered [ 1157.039816] Key type ._llcrypt unregistered [ 1160.449867] Key type ._llcrypt registered [ 1160.451436] Key type .llcrypt registered [ 1160.586472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.598864] alg: No test for adler32 (adler32-zlib) [ 1161.646153] LNetError: 90025:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1161.948376] Key type .llcrypt unregistered [ 1161.949605] Key type ._llcrypt unregistered [ 1165.614951] Key type ._llcrypt registered [ 1165.616423] Key type .llcrypt registered [ 1166.665466] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:01:46 (1760126506) [ 1167.012276] Key type .llcrypt unregistered [ 1167.019213] Key type ._llcrypt unregistered [ 1170.654286] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1170.658963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1173.074229] Key type ._llcrypt registered [ 1173.076381] Key type .llcrypt registered [ 1173.267920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1173.281652] alg: No test for adler32 (adler32-zlib) [ 1174.155427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1174.202792] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1181.903379] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:02:01 (1760126521) [ 1182.018984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1182.041193] LNet: 91791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.051994] LNetError: 91791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1182.076977] LNet: Removed LNI 10.1.2.3@tcp [ 1182.109179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.634931] Key type .llcrypt unregistered [ 1182.636401] Key type ._llcrypt unregistered [ 1186.104809] Key type ._llcrypt registered [ 1186.107913] Key type .llcrypt registered [ 1186.235408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1186.250504] alg: No test for adler32 (adler32-zlib) [ 1187.176094] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1187.228308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 1187.365661] LNet: 92094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.379739] LNetError: 92094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.389814] LNet: Removed LNI 192.168.202.56@tcp [ 1188.884467] Key type .llcrypt unregistered [ 1188.887889] Key type ._llcrypt unregistered [ 1196.354547] Key type ._llcrypt registered [ 1196.356378] Key type .llcrypt registered [ 1197.163492] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:02:16 (1760126536) [ 1197.520285] Key type .llcrypt unregistered [ 1197.522054] Key type ._llcrypt unregistered [ 1201.177532] Key type ._llcrypt registered [ 1201.187170] Key type .llcrypt registered [ 1201.351154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1201.361768] alg: No test for adler32 (adler32-zlib) [ 1206.784721] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:02:26 (1760126546) [ 1206.930668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1206.982365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.034284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1207.053472] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1207.693531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1207.718466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1207.745599] LNet: Removed LNI 192.168.202.56@tcp [ 1207.973734] LNet: 93826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.268397] Key type .llcrypt unregistered [ 1208.271794] Key type ._llcrypt unregistered [ 1216.158750] Key type ._llcrypt registered [ 1216.160432] Key type .llcrypt registered [ 1217.172631] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:02:36 (1760126556) [ 1217.496822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.521471] alg: No test for adler32 (adler32-zlib) [ 1218.384885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.438482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1218.465398] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1219.074808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 1219.112659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.56@tcp [ 1219.138398] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1219.143912] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1219.146907] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1219, 0, 1269 [ 1219.152122] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 1219.155589] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1219.162888] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1219.172536] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1219, 0, 1269 [ 1219.177417] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 1223.919021] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:02:43 (1760126563) [ 1224.262257] LNet: 95359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.271695] LNetError: 95359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.286168] LNet: Removed LNI 192.168.202.56@tcp [ 1224.822938] Key type .llcrypt unregistered [ 1224.824246] Key type ._llcrypt unregistered [ 1229.135968] Key type ._llcrypt registered [ 1229.137251] Key type .llcrypt registered [ 1229.284884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1229.300912] alg: No test for adler32 (adler32-zlib) [ 1230.183346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1230.218910] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1235.824677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 1241.042471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.103784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1241.131381] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 1241.985912] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1242.085546] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1242.088039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1242.090857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1244.328758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1244.341541] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1252.330884] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:03:11 (1760126591) [ 1252.436854] LNet: 96852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.445173] LNetError: 96852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.452710] LNet: Removed LNI 192.168.202.56@tcp [ 1252.456629] LNet: Removed LNI 192.168.202.56@tcp1 [ 1252.472962] LNet: Removed LNI 10.1.2.3@tcp99 [ 1252.931057] Key type .llcrypt unregistered [ 1252.934962] Key type ._llcrypt unregistered [ 1256.995886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1256.998471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1259.371210] Key type ._llcrypt registered [ 1259.372831] Key type .llcrypt registered [ 1259.521072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1259.537305] alg: No test for adler32 (adler32-zlib) [ 1260.410015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.447551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1260.466983] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1261.068552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1261.090569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1262.859205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1262.891776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1262.927347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1262.955411] LNetError: 97588:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1262.976331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1263.008790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 1263.419202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1263.479744] LNet: 97608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.488733] LNetError: 97608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.493872] LNet: Removed LNI 192.168.202.56@tcp [ 1263.506746] LNet: Removed LNI 10.1.2.3@tcp [ 1264.095739] Key type .llcrypt unregistered [ 1264.097483] Key type ._llcrypt unregistered [ 1271.978390] Key type ._llcrypt registered [ 1271.980945] Key type .llcrypt registered [ 1272.861429] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:03:32 (1760126612) [ 1273.142507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.152953] alg: No test for adler32 (adler32-zlib) [ 1273.997179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.025773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1274.040085] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1274.507490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.56@tcp [ 1274.563444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg256-client@tcp [ 1278.820274] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:03:38 (1760126618) [ 1278.913915] LNet: 99126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.918265] LNetError: 99126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.941420] LNet: Removed LNI 192.168.202.56@tcp [ 1279.510876] Key type .llcrypt unregistered [ 1279.517787] Key type ._llcrypt unregistered [ 1283.738147] Key type ._llcrypt registered [ 1283.739651] Key type .llcrypt registered [ 1283.971163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1283.995437] alg: No test for adler32 (adler32-zlib) [ 1284.863256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1284.902250] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1284.936648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1289.977690] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:03:49 (1760126629) [ 1291.241086] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1292.414136] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:03:51 (1760126631) [ 1293.601468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1295.152295] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:03:54 (1760126634) [ 1295.308407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1295.335301] LNet: 100550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.340552] LNetError: 100550:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.383977] LNet: Removed LNI 192.168.202.56@tcp [ 1295.477069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.526076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1295.569374] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1296.150892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1296.182119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1301.780672] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:04:01 (1760126641) [ 1301.905994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.933306] LNet: 101258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.941512] LNetError: 101258:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.975259] LNet: Removed LNI 192.168.202.56@tcp [ 1301.999732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.074886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1302.098466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1302.579603] Key type .llcrypt unregistered [ 1302.583142] Key type ._llcrypt unregistered [ 1309.555482] Key type ._llcrypt registered [ 1309.557285] Key type .llcrypt registered [ 1310.461362] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:04:10 (1760126650) [ 1310.765812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.778869] alg: No test for adler32 (adler32-zlib) [ 1311.647925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.700447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1311.728675] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1312.203911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1312.243632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 1316.474840] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:04:16 (1760126656) [ 1316.534144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1316.553498] LNet: 102801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.558191] LNetError: 102801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.567778] LNet: Removed LNI 192.168.202.56@tcp [ 1316.580118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1316.619763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1316.636864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1321.098658] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1322.244809] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:04:21 (1760126661) [ 1322.314184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.340053] LNet: 103520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.370296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.550603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1322.574711] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1323.065590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1323.080451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1323.653124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.56@tcp [ 1323.685888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1323.719635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.56@tcp -m PUT -e local_timeout -r 1 [ 1323.766293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1323.798973] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1323.802237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1323.958433] LNetError: 103558:0:(lib-move.c:4947:lnet_parse()) 192.168.202.56@tcp, src 192.168.202.56@tcp: Dropping PUT (error -108 looking up sender) [ 1324.960482] LNet: 103835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.967720] LNetError: 103835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.974926] LNet: Removed LNI 192.168.202.56@tcp [ 1324.982928] LNet: Removed LNI 10.1.2.3@tcp [ 1325.242745] Key type .llcrypt unregistered [ 1325.244538] Key type ._llcrypt unregistered [ 1332.574676] Key type ._llcrypt registered [ 1332.576212] Key type .llcrypt registered [ 1333.503054] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:04:32 (1760126672) [ 1333.770487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.783519] alg: No test for adler32 (adler32-zlib) [ 1334.680906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1334.961153] LNet: 104684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.394660] Key type .llcrypt unregistered [ 1335.397246] Key type ._llcrypt unregistered [ 1339.686486] Key type ._llcrypt registered [ 1339.691164] Key type .llcrypt registered [ 1339.836075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.851191] alg: No test for adler32 (adler32-zlib) [ 1340.743946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1340.767276] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1346.909760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 1347.068962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1347.643474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.156@tcp [ 1347.910437] LNet: 105121:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1347.935103] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.156@tcp [ 1347.978725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.007273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1348.490122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1348.742712] LNet: 105226:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1348.798614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.841453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.492278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.156@tcp [ 1349.850704] LNet: 105332:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.898318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.933959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.578899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1351.023483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.045971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.333870] LNet: 105478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.344909] LNetError: 105478:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.364804] LNet: Removed LNI 192.168.202.56@tcp [ 1351.723051] Key type .llcrypt unregistered [ 1351.724850] Key type ._llcrypt unregistered [ 1368.291713] Key type ._llcrypt registered [ 1368.295246] Key type .llcrypt registered [ 1370.066663] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:05:08 (1760126708) [ 1371.045287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1371.078730] alg: No test for adler32 (adler32-zlib) [ 1372.042530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.497303] LNet: 106364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.380507] Key type .llcrypt unregistered [ 1373.381840] Key type ._llcrypt unregistered [ 1381.624593] Key type ._llcrypt registered [ 1381.626027] Key type .llcrypt registered [ 1381.935787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1381.955277] alg: No test for adler32 (adler32-zlib) [ 1382.888040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1382.914530] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1389.811153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.156@tcp [ 1395.793524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.820197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1395.849097] LNet: Added LNI 192.168.202.56@tcp1 [8/256/0/180] [ 1396.594398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1397.041047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.156@tcp [ 1397.284367] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.156@tcp [ 1397.418718] LNet: 107002:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1397.498225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1397.533372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1398.106383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1398.466364] LNet: 107107:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1398.537667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1398.593106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.226194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.156@tcp [ 1399.587093] LNet: 107212:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1399.685943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1399.747589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1400.417695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1400.889669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1400.949847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.362434] LNet: 107393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.367680] LNetError: 107393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.373904] LNet: Removed LNI 192.168.202.56@tcp [ 1405.388227] LNet: Removed LNI 192.168.202.56@tcp1 [ 1405.723814] Key type .llcrypt unregistered [ 1405.725625] Key type ._llcrypt unregistered [ 1418.375442] Key type ._llcrypt registered [ 1418.376837] Key type .llcrypt registered [ 1419.498316] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1206 sec ========== 16:05:58 (1760126758) [ 1420.148655] Key type .llcrypt unregistered [ 1420.152708] Key type ._llcrypt unregistered [ 1430.198440] Key type ._llcrypt registered [ 1430.231037] Key type .llcrypt registered [ 1430.611893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.633453] alg: No test for adler32 (adler32-zlib) [ 1431.618069] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 1431.820464] LNet: Added LNI 192.168.202.56@tcp [8/256/0/180] [ 1433.464251] Key type lgssc registered [ 1434.434484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1483.706273] Lustre: Mounted lustre-client [ 1487.909907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1499.539151] LustreError: 110615:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9a4c206c3000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1499.553462] LustreError: 110615:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1499.588826] Lustre: Unmounted lustre-client [ 1519.421335] Key type lgssc unregistered [ 1519.681860] LNet: 111095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.687762] LNetError: 111095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.706239] LNet: Removed LNI 192.168.202.56@tcp [ 1520.340155] Key type .llcrypt unregistered [ 1520.341485] Key type ._llcrypt unregistered