[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 406228305 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002252] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006877] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007037] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009147] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012045] *** VALIDATE selinux *** [ 0.019668] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023893] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024115] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026088] *** VALIDATE tmpfs *** [ 0.027345] *** VALIDATE proc *** [ 0.028202] *** VALIDATE cgroup *** [ 0.029005] *** VALIDATE cgroup2 *** [ 0.030197] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031113] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033025] Spectre V2 : User space: Vulnerable [ 0.034004] Speculative Store Bypass: Vulnerable [ 0.037429] debug: unmapping init [mem 0xffffffff8ca59000-0xffffffff8ca60fff] [ 0.039237] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040471] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041011] ... version: 2 [ 0.041875] ... bit width: 48 [ 0.042005] ... generic registers: 4 [ 0.042906] ... value mask: 0000ffffffffffff [ 0.043006] ... max period: 00007fffffffffff [ 0.044005] ... fixed-purpose events: 3 [ 0.044831] ... event mask: 000000070000000f [ 0.045224] rcu: Hierarchical SRCU implementation. [ 0.047250] smp: Bringing up secondary CPUs ... [ 0.048392] x86: Booting SMP configuration: [ 0.049011] .... node #0, CPUs: #1 #2 #3 [ 0.052556] smp: Brought up 1 node, 4 CPUs [ 0.054007] smpboot: Max logical packages: 1 [ 0.055010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.277408] node 0 deferred pages initialised in 221ms [ 0.279507] devtmpfs: initialized [ 0.280200] x86/mm: Memory block size: 128MB [ 0.282216] gcov: version magic: 0x41383552 [ 0.283695] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287277] pinctrl core: initialized pinctrl subsystem [ 0.288081] [ 0.288377] ************************************************************* [ 0.290006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291005] ** ** [ 0.292006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.293006] ** ** [ 0.294005] ** This means that this kernel is built to expose internal ** [ 0.296006] ** IOMMU data structures, which may compromise security on ** [ 0.297006] ** your system. ** [ 0.298006] ** ** [ 0.299005] ** If you see this message and you are not debugging the ** [ 0.300010] ** kernel, report this immediately to your vendor! ** [ 0.302006] ** ** [ 0.303005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304005] ************************************************************* [ 0.305535] NET: Registered protocol family 16 [ 0.307296] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.308037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.310027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.312053] cpuidle: using governor menu [ 0.313344] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.314415] PCI: Using configuration type 1 for base access [ 0.316105] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.322094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.324053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.326056] cryptd: max_cpu_qlen set to 1000 [ 0.328223] ACPI: Added _OSI(Module Device) [ 0.329009] ACPI: Added _OSI(Processor Device) [ 0.329999] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.331006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334099] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.338363] ACPI: Interpreter enabled [ 0.339039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.339868] ACPI: Using IOAPIC for interrupt routing [ 0.341049] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343231] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350009] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.353008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.355035] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.357816] acpiphp: Slot [2] registered [ 0.359102] acpiphp: Slot [3] registered [ 0.359961] acpiphp: Slot [4] registered [ 0.361042] acpiphp: Slot [5] registered [ 0.361866] acpiphp: Slot [6] registered [ 0.362057] acpiphp: Slot [7] registered [ 0.362842] acpiphp: Slot [8] registered [ 0.364040] acpiphp: Slot [9] registered [ 0.364861] acpiphp: Slot [10] registered [ 0.365041] acpiphp: Slot [11] registered [ 0.365818] acpiphp: Slot [12] registered [ 0.367043] acpiphp: Slot [13] registered [ 0.368045] acpiphp: Slot [14] registered [ 0.368876] acpiphp: Slot [15] registered [ 0.370042] acpiphp: Slot [16] registered [ 0.370996] acpiphp: Slot [17] registered [ 0.372043] acpiphp: Slot [18] registered [ 0.372863] acpiphp: Slot [19] registered [ 0.373044] acpiphp: Slot [20] registered [ 0.373873] acpiphp: Slot [21] registered [ 0.375041] acpiphp: Slot [22] registered [ 0.375896] acpiphp: Slot [23] registered [ 0.376045] acpiphp: Slot [24] registered [ 0.376875] acpiphp: Slot [25] registered [ 0.378043] acpiphp: Slot [26] registered [ 0.378880] acpiphp: Slot [27] registered [ 0.380069] acpiphp: Slot [28] registered [ 0.381063] acpiphp: Slot [29] registered [ 0.381915] acpiphp: Slot [30] registered [ 0.383044] acpiphp: Slot [31] registered [ 0.383860] PCI host bridge to bus 0000:00 [ 0.384012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.391021] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.393030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.394184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.397674] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.399558] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.404386] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.407535] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.410007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.412008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414259] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.416773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.418021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.420380] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.421986] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.429723] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.432008] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435980] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.446013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.457020] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.466759] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.475009] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.485013] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.492528] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.493323] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.495202] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.496236] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.498110] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.502165] iommu: Default domain type: Passthrough [ 0.504559] SCSI subsystem initialized [ 0.506133] ACPI: bus type USB registered [ 0.507128] usbcore: registered new interface driver usbfs [ 0.510057] usbcore: registered new interface driver hub [ 0.512061] usbcore: registered new device driver usb [ 0.513208] pps_core: LinuxPPS API ver. 1 registered [ 0.515008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.519045] PTP clock support registered [ 0.521105] EDAC MC: Ver: 3.0.0 [ 0.523148] PCI: Using ACPI for IRQ routing [ 0.524695] NetLabel: Initializing [ 0.525006] NetLabel: domain hash size = 128 [ 0.525830] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.527039] NetLabel: unlabeled traffic allowed by default [ 0.528153] vgaarb: loaded [ 0.529335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.531012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.536384] clocksource: Switched to clocksource kvm-clock [ 0.668313] VFS: Disk quotas dquot_6.6.0 [ 0.669598] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.671927] *** VALIDATE ramfs *** [ 0.672740] *** VALIDATE hugetlbfs *** [ 0.674374] pnp: PnP ACPI init [ 0.675957] pnp: PnP ACPI: found 6 devices [ 0.691121] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.694501] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.696787] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.698931] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.701460] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.703918] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.706748] NET: Registered protocol family 2 [ 0.708838] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.712862] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.715653] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.720301] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.722928] TCP: Hash tables configured (established 65536 bind 65536) [ 0.725050] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.727462] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.729656] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.732071] NET: Registered protocol family 1 [ 0.734212] RPC: Registered named UNIX socket transport module. [ 0.736009] RPC: Registered udp transport module. [ 0.737411] RPC: Registered tcp transport module. [ 0.738940] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.740826] NET: Registered protocol family 44 [ 0.742068] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.743716] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.745153] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.746781] PCI: CLS 0 bytes, default 64 [ 0.747921] Unpacking initramfs... [ 2.110075] debug: unmapping init [mem 0xffff9e8bfcc64000-0xffff9e8bfffcffff] [ 2.112534] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.113697] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.115920] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.652806] Initialise system trusted keyrings [ 2.654646] Key type blacklist registered [ 2.656658] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.665686] zbud: loaded [ 2.668714] *** VALIDATE nfs *** [ 2.670184] *** VALIDATE nfs4 *** [ 2.671502] pstore: using deflate compression [ 2.673144] Platform Keyring initialized [ 2.756631] NET: Registered protocol family 38 [ 2.758430] Key type asymmetric registered [ 2.759910] Asymmetric key parser 'x509' registered [ 2.761667] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.764522] io scheduler mq-deadline registered [ 2.766157] io scheduler kyber registered [ 2.767738] io scheduler bfq registered [ 2.769397] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.772117] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.774842] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.777554] ACPI: Power Button [PWRF] [ 2.879590] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.970136] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.046394] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.077595] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.106811] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.113574] Non-volatile memory driver v1.3 [ 3.114778] Linux agpgart interface v0.103 [ 3.144677] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.147065] vda: detected capacity change from 0 to 68222976 [ 3.159519] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.162254] vdb: detected capacity change from 0 to 1073741824 [ 3.170983] libphy: Fixed MDIO Bus: probed [ 3.176241] usbcore: registered new interface driver usbserial_generic [ 3.178118] usbserial: USB Serial support registered for generic [ 3.179834] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.184337] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.185755] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.187555] mousedev: PS/2 mouse device common for all mice [ 3.189678] rtc_cmos 00:05: RTC can wake from S4 [ 3.192309] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.196485] rtc_cmos 00:05: registered as rtc0 [ 3.197960] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.198896] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.202792] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.204384] intel_pstate: CPU model not supported [ 3.209430] hid: raw HID events driver (C) Jiri Kosina [ 3.211678] usbcore: registered new interface driver usbhid [ 3.213412] usbhid: USB HID core driver [ 3.214764] drop_monitor: Initializing network drop monitor service [ 3.216772] Initializing XFRM netlink socket [ 3.218099] NET: Registered protocol family 10 [ 3.221377] Segment Routing with IPv6 [ 3.222280] NET: Registered protocol family 17 [ 3.223436] mpls_gso: MPLS GSO support [ 3.227731] RAS: Correctable Errors collector initialized. [ 3.229817] AVX version of gcm_enc/dec engaged. [ 3.231369] AES CTR mode by8 optimization enabled [ 3.303561] sched_clock: Marking stable (3303538547, 0)->(3972854668, -669316121) [ 3.306541] registered taskstats version 1 [ 3.307814] Loading compiled-in X.509 certificates [ 3.309015] zswap: loaded using pool lzo/zbud [ 3.329602] Key type big_key registered [ 3.341543] Key type encrypted registered [ 3.342870] ima: No TPM chip found, activating TPM-bypass! [ 3.344492] ima: Allocated hash algorithm: sha1 [ 3.345768] ima: No architecture policies found [ 3.347123] evm: Initialising EVM extended attributes: [ 3.348543] evm: security.selinux [ 3.349414] evm: security.ima [ 3.350238] evm: security.capability [ 3.351232] evm: HMAC attrs: 0x1 [ 3.353269] rtc_cmos 00:05: setting system clock to 2025-10-10 23:06:05 UTC (1760137565) [ 3.358720] debug: unmapping init [mem 0xffffffff8da03000-0xffffffff8dbfffff] [ 3.361497] debug: unmapping init [mem 0xffffffff8c782000-0xffffffff8ca58fff] [ 3.369819] Write protecting the kernel read-only data: 28672k [ 3.373359] debug: unmapping init [mem 0xffffffff8ae03000-0xffffffff8affffff] [ 3.376165] debug: unmapping init [mem 0xffffffff8b714000-0xffffffff8b7fffff] [ 3.405132] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.413394] systemd[1]: Detected virtualization kvm. [ 3.415364] systemd[1]: Detected architecture x86-64. [ 3.417152] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.447218] systemd[1]: No hostname configured. [ 3.448727] systemd[1]: Set hostname to . [ 3.450365] random: systemd: uninitialized urandom read (16 bytes read) [ 3.452358] systemd[1]: Initializing machine ID from random generator. [ 3.497724] random: ln: uninitialized urandom read (6 bytes read) [ 3.582461] random: systemd: uninitialized urandom read (16 bytes read) [ 3.584487] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.591223] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.599618] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.161102] device-mapper: uevent: version 1.0.3 [ 4.162685] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.914720] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.978149] scsi host0: ata_piix [ 4.981720] scsi host1: ata_piix [ 4.983024] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.985450] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.819609] random: fast init done [ 9.597614] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.856571] random: crng init done [ 9.857780] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.418979] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.771713] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.249032] SELinux: Disabled at runtime. [ 12.318307] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.332707] systemd[1]: Detected virtualization kvm. [ 12.334411] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.038542] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.040576] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.043587] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.046145] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.048372] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.053389] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.060929] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 13.131844] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.434570] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.781734] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.849701] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.970268] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.992570] EDAC sbridge: Ver: 1.1.2 [ 15.034338] Key type dns_resolver registered [ 15.396252] NFS: Registering the id_resolver key type [ 15.398268] Key type id_resolver registered [ 15.399705] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg204-client login: [ 44.108705] libcfs: loading out-of-tree module taints kernel. [ 44.146401] Key type ._llcrypt registered [ 44.155567] Key type .llcrypt registered [ 44.324711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.329057] alg: No test for adler32 (adler32-zlib) [ 45.287849] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 45.567165] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 47.167130] Key type lgssc registered [ 47.684281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.669128] Lustre: Mounted lustre-client [ 93.961319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 106.748397] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing check_logdir /tmp/testlogs/ [ 108.187087] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing yml_node [ 109.598208] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 110.707761] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 111.860638] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 112.615244] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Oct 10 19:07:54 EDT 2025 [ 117.215273] Lustre: lustre-OST0000-osc-ffff9e8c50e3f800: disconnect after 24s idle [ 118.668900] LustreError: 7912:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e8c50e3f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 118.672739] LustreError: 7912:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 118.684301] Lustre: Unmounted lustre-client [ 130.217056] Key type lgssc unregistered [ 130.337378] LNet: 8357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 130.340995] LNetError: 8357:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 130.348741] LNet: Removed LNI 192.168.202.4@tcp [ 130.605089] Key type .llcrypt unregistered [ 130.606188] Key type ._llcrypt unregistered [ 139.109753] Key type ._llcrypt registered [ 139.110741] Key type .llcrypt registered [ 139.170538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 139.175188] alg: No test for adler32 (adler32-zlib) [ 139.973103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 139.995911] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 140.002903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 141.289354] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 141.321158] LNet: 9221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 141.324720] LNetError: 9221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 141.337599] LNet: Removed LNI 192.168.202.4@tcp [ 141.546235] Key type .llcrypt unregistered [ 141.547149] Key type ._llcrypt unregistered [ 143.461670] Key type ._llcrypt registered [ 143.463055] Key type .llcrypt registered [ 143.940322] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:08:25 (1760137705) [ 144.047519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 144.053677] alg: No test for adler32 (adler32-zlib) [ 144.859872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 144.883898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 144.891460] LNet: 9634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 144.897883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 144.917142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 144.923432] LNet: 9641:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 146.950888] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:08:28 (1760137708) [ 146.977576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.986112] LNet: 10213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 146.995238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 147.024096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 149.007648] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:08:30 (1760137710) [ 149.037941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 149.045673] LNet: 10797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 149.054808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 149.082225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 151.138716] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:08:32 (1760137712) [ 151.219431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 151.226142] LNet: 11384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 151.233400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.257350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 153.248082] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:08:34 (1760137714) [ 153.304397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 153.315422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 153.336143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 153.455703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 155.695354] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:08:37 (1760137717) [ 155.742759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 155.752908] LNet: 12554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 155.755452] LNet: 12554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 155.770325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 155.801882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 155.934883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 156.071886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 158.258666] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:08:39 (1760137719) [ 158.294316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 158.324048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 158.359141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 160.522901] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:08:42 (1760137722) [ 160.556690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 160.575796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 160.602758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 160.613063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 160.722246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 160.733330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 160.839545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 160.850151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 160.958896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 160.969601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 161.078066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 161.089783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 161.200566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 161.211419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 161.320279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 161.330374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 161.438529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 161.448949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 161.559321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 161.570120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 161.690537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 161.704777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 161.837193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 161.851952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 161.970961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 161.983607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 162.105186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 162.116636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 164.073881] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:08:45 (1760137725) [ 164.110449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 164.118760] LNet: 14414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 164.120767] LNet: 14414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 164.129526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 164.157183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 164.167995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 166.121523] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:08:47 (1760137727) [ 166.169864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 166.187843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 166.214695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 166.225453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 168.209812] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:08:49 (1760137729) [ 168.291399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.303156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.326964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 168.339531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 170.377675] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:08:52 (1760137732) [ 170.416982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.431318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.459069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 170.471310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 172.590535] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:08:54 (1760137734) [ 172.631218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.648535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.674275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 172.688994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 174.757438] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:08:56 (1760137736) [ 174.785740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.803519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.831522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 174.843478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 177.015678] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:08:58 (1760137738) [ 177.046969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 177.064930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 177.108460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 177.253485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 179.451472] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:09:01 (1760137741) [ 179.477674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.496353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.569553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 179.644038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 181.882110] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:09:03 (1760137743) [ 181.916909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.926782] LNet: 19138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 181.928626] LNet: 19138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 181.939844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.999293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 182.083920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 184.248218] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:09:05 (1760137745) [ 184.295524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.311522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.440258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 184.529271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 186.731219] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:09:08 (1760137748) [ 186.770393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.792503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.825595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 186.846099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 189.086898] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:09:10 (1760137750) [ 189.120498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.138383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.169243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 189.187471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 191.411675] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:09:13 (1760137753) [ 191.435302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.449983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.476554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 193.714258] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:09:15 (1760137755) [ 193.747909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.764912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.800792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 196.004362] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:09:17 (1760137757) [ 196.036229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.053886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.085289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 196.213275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 198.317811] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:09:20 (1760137760) [ 198.364462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.385656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.414389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 198.430462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 200.596973] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:09:22 (1760137762) [ 200.644397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.662988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.698026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 200.710203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 202.878225] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:09:24 (1760137764) [ 202.914104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.935318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.964700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 202.980056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 205.122839] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:09:26 (1760137766) [ 205.155778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.172061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.212041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 205.227595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 207.369206] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:09:29 (1760137769) [ 207.402953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.426897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.454652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 207.479019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 207.487469] LNetError: 25717:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 207.501262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 207.513563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 209.512423] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:09:31 (1760137771) [ 209.541912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.557116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.589156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PPL1WebcX7 [ 209.616791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PPL1WebcX7 [ 209.634977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.PPL1WebcX7 [ 211.571448] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:09:33 (1760137773) [ 211.601709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.621478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.645744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 211.658940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 211.674424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 213.614649] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:09:35 (1760137775) [ 213.649876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.668410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.689629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 214.673957] LNet: Message forwarding will be enabled in 2s [ 217.759187] LNet: Message forwarding enabled [ 217.818716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 217.830046] LNet: Message forwarding will stop in 2s to 6s [ 221.975387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 222.551677] LNet: Message forwarding disabled [ 222.975387] LNet: Message forwarding will be enabled in 2s [ 224.946520] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:09:46 (1760137786) [ 224.970840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.979778] LNet: 28154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 224.982533] LNet: 28154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 225.527657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.553156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 227.526133] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:09:49 (1760137789) [ 227.562448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.577982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.604318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 227.617276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 227.628820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 227.639312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 227.650525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 227.662739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 227.673593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 227.688517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 227.708285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 227.721642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 227.734334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 227.749073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 227.763681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 227.775896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 227.787143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 227.799556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 227.816274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 227.829097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 227.840316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 227.847339] LNetError: 28782:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 229.764657] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:09:51 (1760137791) [ 229.794069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.811372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.845456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 231.926946] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:09:53 (1760137793) [ 232.507848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg204-server [ 233.091899] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:09:54 (1760137794) [ 233.130448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.147426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.166143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 233.173260] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 233.691527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[5-9]@tcp [ 233.700601] LNet: 30493:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 233.708316] LNetError: 30286:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 233.713984] LNetError: 30286:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.7@tcp has gone from up to down [ 234.720699] LNetError: 30286:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 234.725622] LNetError: 30286:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 235.744572] LNetError: 30286:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 235.748257] LNetError: 30286:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 235.953331] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:09:57 (1760137797) [ 236.632964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg204-server [ 236.767688] LNetError: 30292:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.9@tcp has gone from up to down [ 236.770968] LNetError: 30292:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 237.243613] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:09:58 (1760137798) [ 237.294493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.304828] LNetError: 31428:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 237.312042] LNet: Removed LNI 192.168.202.4@tcp [ 237.319618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 237.337310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 237.344591] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 237.583576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[5-10/2]@tcp [ 237.601314] LNet: 31542:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 237.608525] LNet: 31542:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 237.620797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[5-10/2]@tcp [ 239.842853] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:10:01 (1760137801) [ 239.877239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.885806] LNetError: 32129:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 239.894500] LNet: Removed LNI 192.168.202.4@tcp [ 239.903363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.943726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 239.970446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.984140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.010512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 240.034329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.046417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.063889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 240.090944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.102644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.124486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 240.150995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.162966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.179574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 240.213301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.230196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.257993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 240.286700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.298415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.320434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 240.349176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.366570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.390934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 240.419918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.431290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.447759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 240.477801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.490237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.510603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 240.536229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.549502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.567207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 240.587280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.598798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 240.620097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 242.638452] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:10:04 (1760137804) [ 243.271241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg204-server [ 243.829327] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:10:05 (1760137805) [ 244.403428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg204-server [ 244.935488] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:10:06 (1760137806) [ 245.199127] Key type .llcrypt unregistered [ 245.200932] Key type ._llcrypt unregistered [ 247.045449] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 247.113594] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 247.114937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 247.118133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 249.339742] Key type ._llcrypt registered [ 249.340903] Key type .llcrypt registered [ 249.402727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 249.407908] alg: No test for adler32 (adler32-zlib) [ 250.202826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.219669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 250.229080] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 250.428588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 250.436461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 251.014852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 251.021896] LNet: Removed LNI 10.1.2.3@tcp [ 251.057530] LNet: 34241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 251.059755] LNetError: 34241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 251.069519] LNet: Removed LNI 192.168.202.4@tcp [ 251.279358] Key type .llcrypt unregistered [ 251.280416] Key type ._llcrypt unregistered [ 254.615601] Key type ._llcrypt registered [ 254.616580] Key type .llcrypt registered [ 255.078718] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:10:16 (1760137816) [ 255.249449] Key type .llcrypt unregistered [ 255.250715] Key type ._llcrypt unregistered [ 257.042652] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 257.075445] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 257.079042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 257.080537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 259.278416] Key type ._llcrypt registered [ 259.279537] Key type .llcrypt registered [ 259.339449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 259.344384] alg: No test for adler32 (adler32-zlib) [ 260.148199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.169845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 260.178989] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 260.410763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 260.423570] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 261.308229] LNet: 35719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.311224] LNetError: 35719:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.313222] LNet: Removed LNI 192.168.202.4@tcp [ 261.322346] LNet: Removed LNI 10.1.2.3@tcp [ 261.538249] Key type .llcrypt unregistered [ 261.539328] Key type ._llcrypt unregistered [ 265.363744] Key type ._llcrypt registered [ 265.365231] Key type .llcrypt registered [ 265.949744] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:10:27 (1760137827) [ 266.167325] Key type .llcrypt unregistered [ 266.168594] Key type ._llcrypt unregistered [ 268.193798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 268.253895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 268.255564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 268.257611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 270.537958] Key type ._llcrypt registered [ 270.539159] Key type .llcrypt registered [ 270.621040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 270.628167] alg: No test for adler32 (adler32-zlib) [ 271.427920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.446182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 271.454488] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 272.095984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 272.104608] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 272.737378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 272.745024] LNet: Removed LNI 10.1.2.3@tcp [ 272.750562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 272.769297] LNet: Removed LNI 10.1.2.31@tcp [ 272.775567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 272.785677] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 273.021787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 273.029622] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 273.350442] LNet: 37617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.353793] LNetError: 37617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 273.357297] LNet: Removed LNI 10.1.2.3@tcp [ 273.371444] LNet: Removed LNI 10.1.2.31@tcp [ 273.584331] Key type .llcrypt unregistered [ 273.585834] Key type ._llcrypt unregistered [ 277.293969] Key type ._llcrypt registered [ 277.294967] Key type .llcrypt registered [ 277.795642] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:10:39 (1760137839) [ 277.945173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 277.951982] alg: No test for adler32 (adler32-zlib) [ 278.749834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 278.770727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 278.778653] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 279.014323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.023715] LNet: 38561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.027243] LNetError: 38561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.037294] LNet: Removed LNI 192.168.202.4@tcp [ 279.046142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.065310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.4@tcp [ 279.073574] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 279.291842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.4@tcp [ 279.311349] LNet: Removed LNI 192.168.202.4@tcp [ 279.567303] Key type .llcrypt unregistered [ 279.568450] Key type ._llcrypt unregistered [ 282.914894] Key type ._llcrypt registered [ 282.915874] Key type .llcrypt registered [ 283.330444] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:10:45 (1760137845) [ 283.866895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg204-server [ 284.347323] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:10:46 (1760137846) [ 284.503273] Key type .llcrypt unregistered [ 284.504231] Key type ._llcrypt unregistered [ 286.313147] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 286.352330] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 286.354490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 286.355905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 288.545312] Key type ._llcrypt registered [ 288.546410] Key type .llcrypt registered [ 288.602294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 288.607119] alg: No test for adler32 (adler32-zlib) [ 289.400409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.417371] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 289.609153] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 290.256546] LNet: 40513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.259247] LNetError: 40513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.264206] LNet: Removed LNI 192.168.202.4@tcp1 [ 290.272319] LNet: Removed LNI 10.1.2.3@tcp2 [ 290.483408] Key type .llcrypt unregistered [ 290.484654] Key type ._llcrypt unregistered [ 294.050071] Key type ._llcrypt registered [ 294.051147] Key type .llcrypt registered [ 294.481021] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:10:56 (1760137856) [ 294.588203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 294.593993] alg: No test for adler32 (adler32-zlib) [ 295.400119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.418108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 295.425888] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 295.654950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.661552] LNet: 41469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.663864] LNetError: 41469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.670213] LNet: Removed LNI 192.168.202.4@tcp [ 295.675687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.712391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 295.723191] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 297.769712] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:10:59 (1760137859) [ 297.797085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.805131] LNet: 42059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.807170] LNetError: 42059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 297.815643] LNet: Removed LNI 192.168.202.4@tcp [ 297.825267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.848219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 297.856750] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 298.131441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.4 [ 298.139870] LNet: Removed LNI 192.168.202.4@tcp [ 300.156416] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:11:01 (1760137861) [ 300.191404] LNet: 42758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.193673] LNetError: 42758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 300.201436] LNet: Removed LNI 192.168.202.4@tcp [ 300.429427] Key type .llcrypt unregistered [ 300.431038] Key type ._llcrypt unregistered [ 302.290933] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 302.409058] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 302.411368] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 302.413496] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 304.680532] Key type ._llcrypt registered [ 304.682382] Key type .llcrypt registered [ 304.748322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 304.755692] alg: No test for adler32 (adler32-zlib) [ 305.688658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 307.657513] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:11:09 (1760137869) [ 307.699832] LNet: 43662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.702196] LNetError: 43662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 307.712709] LNet: Removed LNI 10.1.2.3@tcp [ 307.926330] Key type .llcrypt unregistered [ 307.927470] Key type ._llcrypt unregistered [ 309.777307] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 309.862340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 309.863979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 309.865393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 312.107046] Key type ._llcrypt registered [ 312.108196] Key type .llcrypt registered [ 312.162631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.170717] alg: No test for adler32 (adler32-zlib) [ 312.983897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 314.939671] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:11:16 (1760137876) [ 314.969567] LNet: 44544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.971926] LNetError: 44544:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 314.981519] LNet: Removed LNI 10.1.2.3@tcp [ 315.221456] Key type .llcrypt unregistered [ 315.222496] Key type ._llcrypt unregistered [ 316.969939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 317.058578] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 317.060230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 317.061824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 319.335045] Key type ._llcrypt registered [ 319.336028] Key type .llcrypt registered [ 319.389389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.394393] alg: No test for adler32 (adler32-zlib) [ 320.201501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 322.074550] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:11:23 (1760137883) [ 322.121848] LNet: 45437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.124068] LNetError: 45437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 322.132865] LNet: Removed LNI 10.1.2.3@tcp [ 322.360442] Key type .llcrypt unregistered [ 322.362275] Key type ._llcrypt unregistered [ 324.179425] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 324.290565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 324.292324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 324.293895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 326.541623] Key type ._llcrypt registered [ 326.542804] Key type .llcrypt registered [ 326.599965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 326.606739] alg: No test for adler32 (adler32-zlib) [ 327.404465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.429889] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 330.178209] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:11:31 (1760137891) [ 330.302866] LNet: 46658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.305219] LNetError: 46658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 330.316476] LNet: Removed LNI 10.1.2.3@tcp [ 330.531303] Key type .llcrypt unregistered [ 330.532441] Key type ._llcrypt unregistered [ 332.290573] Key type ._llcrypt registered [ 332.291722] Key type .llcrypt registered [ 332.346813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 332.351687] alg: No test for adler32 (adler32-zlib) [ 333.151826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 333.161709] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 335.896723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 336.110778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_interrupt -r 1 [ 336.124133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_interrupt -r 1 [ 336.139095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 336.145534] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 336.253126] LNet: 47082:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 336.285229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 336.297721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 336.308453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 336.465497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_dropped -r 1 [ 336.477347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_dropped -r 1 [ 336.488818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 336.620947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 336.632604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 336.643630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 336.808426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_aborted -r 1 [ 336.823368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_aborted -r 1 [ 336.835785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 336.845705] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 336.847575] LNet: Skipped 1 previous similar message [ 336.953626] LNet: 47277:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 336.955994] LNet: 47277:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 336.979103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 336.990306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 337.002420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 337.166329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_no_route -r 1 [ 337.180462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_no_route -r 1 [ 337.196386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 337.317431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 337.330450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 337.342179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 337.508695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_timeout -r 1 [ 337.521058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_timeout -r 1 [ 337.533739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 337.659798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 337.673578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 337.685143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 337.857576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_error -r 1 [ 337.870242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_error -r 1 [ 337.883921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 337.891274] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 337.893569] LNet: Skipped 2 previous similar messages [ 337.993378] LNet: 47568:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 337.995437] LNet: 47568:0:(api-ni.c:339:recovery_interval_set()) Skipped 2 previous similar messages [ 338.017244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 338.029047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 338.042588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 338.130294] LNet: 47604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.132555] LNetError: 47604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.143358] LNet: Removed LNI 192.168.202.4@tcp [ 338.293444] Key type .llcrypt unregistered [ 338.294877] Key type ._llcrypt unregistered [ 343.132653] Key type ._llcrypt registered [ 343.133668] Key type .llcrypt registered [ 343.546150] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:11:45 (1760137905) [ 343.852269] Key type .llcrypt unregistered [ 343.853301] Key type ._llcrypt unregistered [ 345.696961] Key type ._llcrypt registered [ 345.698169] Key type .llcrypt registered [ 345.770221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.776669] alg: No test for adler32 (adler32-zlib) [ 346.578288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 346.588431] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 349.276735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 352.534148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.545226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 352.552135] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 353.004440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_interrupt -r 1 [ 353.019562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_interrupt -r 1 [ 353.034312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_interrupt -r 1 [ 353.045853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_interrupt -r 1 [ 353.058256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_interrupt -r 1 [ 353.071381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_interrupt -r 1 [ 353.085464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_interrupt -r 1 [ 353.096951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_interrupt -r 1 [ 353.109613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 353.116282] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 353.119202] LNet: 1 local NIs in recovery (showing 1): 192.168.202.4@tcp [ 353.219367] LNet: 49056:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 353.244663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 353.258641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 353.270952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 353.442298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_dropped -r 1 [ 353.453912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_dropped -r 1 [ 353.464983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_dropped -r 1 [ 353.475930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_dropped -r 1 [ 353.487134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_dropped -r 1 [ 353.498871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_dropped -r 1 [ 353.510379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_dropped -r 1 [ 353.522697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_dropped -r 1 [ 353.536468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 353.664287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 353.677419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 353.689400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 353.855113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_aborted -r 1 [ 353.866485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_aborted -r 1 [ 353.877409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_aborted -r 1 [ 353.888214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_aborted -r 1 [ 353.899959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_aborted -r 1 [ 353.911839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_aborted -r 1 [ 353.923373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_aborted -r 1 [ 353.935898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_aborted -r 1 [ 353.948104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 353.955665] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 353.957794] LNet: Skipped 5 previous similar messages [ 354.063984] LNet: 49274:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 354.068600] LNet: 49274:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 354.097976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 354.110776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 354.122404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 354.288306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_no_route -r 1 [ 354.301839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_no_route -r 1 [ 354.316327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_no_route -r 1 [ 354.327987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_no_route -r 1 [ 354.340848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_no_route -r 1 [ 354.357512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_no_route -r 1 [ 354.372953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_no_route -r 1 [ 354.387163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_no_route -r 1 [ 354.403440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 354.520029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 354.531352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 354.542723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 354.744351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_timeout -r 1 [ 354.762775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_timeout -r 1 [ 354.775783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_timeout -r 1 [ 354.789454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_timeout -r 1 [ 354.805786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_timeout -r 1 [ 354.821836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_timeout -r 1 [ 354.836475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_timeout -r 1 [ 354.848732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_timeout -r 1 [ 354.862932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 355.017484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 355.033825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 355.049939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 355.222914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_error -r 1 [ 355.235159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_error -r 1 [ 355.246451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_error -r 1 [ 355.257840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_error -r 1 [ 355.270207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_error -r 1 [ 355.283398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_error -r 1 [ 355.295368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_error -r 1 [ 355.307083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_error -r 1 [ 355.322512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 355.332931] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 355.334752] LNet: Skipped 9 previous similar messages [ 355.435187] LNet: 49601:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 355.437327] LNet: 49601:0:(api-ni.c:339:recovery_interval_set()) Skipped 2 previous similar messages [ 355.457912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 355.470828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 355.482103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 357.568747] LNet: 49672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.571469] LNetError: 49672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.574653] LNet: Removed LNI 192.168.202.4@tcp [ 358.629553] LNet: Removed LNI 192.168.202.4@tcp1 [ 358.796655] Key type .llcrypt unregistered [ 358.797728] Key type ._llcrypt unregistered [ 364.082142] Key type ._llcrypt registered [ 364.083289] Key type .llcrypt registered [ 364.508413] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:12:06 (1760137926) [ 364.769431] Key type .llcrypt unregistered [ 364.770555] Key type ._llcrypt unregistered [ 366.622327] Key type ._llcrypt registered [ 366.624070] Key type .llcrypt registered [ 366.697746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 366.703339] alg: No test for adler32 (adler32-zlib) [ 367.511860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 367.520848] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 370.339077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 370.537566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_dropped -r 1 [ 370.549609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_dropped -r 1 [ 370.562873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 370.571111] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 370.681109] LNet: 50919:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 370.704559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 370.715637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 370.726419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 370.927101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_error -r 1 [ 370.939854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_error -r 1 [ 370.952479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 371.090049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 371.105105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 371.117519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 371.319044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_timeout -r 1 [ 371.335650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_timeout -r 1 [ 371.349595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 371.359134] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 371.362039] LNet: Skipped 1 previous similar message [ 371.478633] LNet: 51113:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 371.481044] LNet: 51113:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 371.504816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 371.516672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 371.531990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 371.635241] LNet: 51149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.639256] LNetError: 51149:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 371.652489] LNet: Removed LNI 192.168.202.4@tcp [ 371.822398] Key type .llcrypt unregistered [ 371.824224] Key type ._llcrypt unregistered [ 376.856660] Key type ._llcrypt registered [ 376.857610] Key type .llcrypt registered [ 377.312218] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:12:18 (1760137938) [ 377.573182] Key type .llcrypt unregistered [ 377.574293] Key type ._llcrypt unregistered [ 379.432279] Key type ._llcrypt registered [ 379.433297] Key type .llcrypt registered [ 379.493979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 379.499092] alg: No test for adler32 (adler32-zlib) [ 380.298058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 380.307154] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 383.209776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 385.652771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.664776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 385.672735] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 386.179979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_dropped -r 1 [ 386.199512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e remote_dropped -r 1 [ 386.214302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_dropped -r 1 [ 386.230669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e remote_dropped -r 1 [ 386.245319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e remote_dropped -r 1 [ 386.262030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e remote_dropped -r 1 [ 386.275990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e remote_dropped -r 1 [ 386.290632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e remote_dropped -r 1 [ 386.305119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 386.311923] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 386.315605] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.104@tcp [ 386.443465] LNet: 52595:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 386.470357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 386.483638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 386.495843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 386.507538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 386.690472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_error -r 1 [ 386.702541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e remote_error -r 1 [ 386.715467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_error -r 1 [ 386.728267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e remote_error -r 1 [ 386.745207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e remote_error -r 1 [ 386.763648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e remote_error -r 1 [ 386.779691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e remote_error -r 1 [ 386.792366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e remote_error -r 1 [ 386.806436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 386.814727] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 386.816470] LNet: Skipped 2 previous similar messages [ 386.951714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 386.965288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 386.980274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 386.996533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 387.205873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_timeout -r 1 [ 387.225698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e remote_timeout -r 1 [ 387.242807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_timeout -r 1 [ 387.258206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e remote_timeout -r 1 [ 387.276173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e remote_timeout -r 1 [ 387.289904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e remote_timeout -r 1 [ 387.305688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e remote_timeout -r 1 [ 387.323505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e remote_timeout -r 1 [ 387.341617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.104@tcp [ 387.478064] LNet: 52817:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 387.481356] LNet: 52817:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 387.508697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 387.522124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 387.535022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 387.549355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 389.567714] LNet: 52889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.569828] LNetError: 52889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.572040] LNet: Removed LNI 192.168.202.4@tcp [ 389.577490] LNet: Removed LNI 192.168.202.4@tcp1 [ 389.744761] Key type .llcrypt unregistered [ 389.746078] Key type ._llcrypt unregistered [ 395.135129] Key type ._llcrypt registered [ 395.136627] Key type .llcrypt registered [ 395.606662] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:12:37 (1760137957) [ 395.768340] Key type .llcrypt unregistered [ 395.769968] Key type ._llcrypt unregistered [ 397.775470] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 397.841163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 397.843814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 397.846462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 400.117634] Key type ._llcrypt registered [ 400.119959] Key type .llcrypt registered [ 400.204407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 400.211538] alg: No test for adler32 (adler32-zlib) [ 401.022344] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 401.035271] LNet: 53968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.037364] LNetError: 53968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 401.047413] LNet: Removed LNI 192.168.202.4@tcp [ 401.315280] Key type .llcrypt unregistered [ 401.316469] Key type ._llcrypt unregistered [ 403.372242] Key type ._llcrypt registered [ 403.373313] Key type .llcrypt registered [ 403.437690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 403.443812] alg: No test for adler32 (adler32-zlib) [ 404.253256] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 404.254717] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 404.266845] LNet: 54237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.270872] LNetError: 54237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 404.275487] LNet: Removed LNI 192.168.202.4@tcp [ 404.283504] LNet: Removed LNI 10.1.2.3@tcp1 [ 404.568145] Key type .llcrypt unregistered [ 404.569126] Key type ._llcrypt unregistered [ 406.610686] Key type ._llcrypt registered [ 406.611893] Key type .llcrypt registered [ 406.669539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 406.674264] alg: No test for adler32 (adler32-zlib) [ 407.490543] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 407.492887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 407.506423] LNet: 54505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.508700] LNetError: 54505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 407.510655] LNet: Removed LNI 192.168.202.4@tcp [ 407.516432] LNet: Removed LNI 10.1.2.3@tcp [ 407.812455] Key type .llcrypt unregistered [ 407.813570] Key type ._llcrypt unregistered [ 409.816097] Key type ._llcrypt registered [ 409.817143] Key type .llcrypt registered [ 409.879182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 409.884926] alg: No test for adler32 (adler32-zlib) [ 410.698791] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 410.700343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 410.712537] LNet: 54775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.714365] LNetError: 54775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 410.716219] LNet: Removed LNI 192.168.202.4@tcp [ 410.725539] LNet: Removed LNI 10.1.2.3@tcp [ 410.980491] Key type .llcrypt unregistered [ 410.982765] Key type ._llcrypt unregistered [ 412.792347] Key type ._llcrypt registered [ 412.793275] Key type .llcrypt registered [ 412.847319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 412.852484] alg: No test for adler32 (adler32-zlib) [ 413.661930] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 413.663816] LNetError: 55035:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 413.666278] LNetError: Error -100 starting up LNI tcp [ 413.673327] LNet: Removed LNI 192.168.202.4@tcp [ 413.921389] Key type .llcrypt unregistered [ 413.922389] Key type ._llcrypt unregistered [ 417.659221] Key type ._llcrypt registered [ 417.661061] Key type .llcrypt registered [ 418.157667] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:12:59 (1760137979) [ 418.472287] Key type .llcrypt unregistered [ 418.473775] Key type ._llcrypt unregistered [ 420.364901] Key type ._llcrypt registered [ 420.365779] Key type .llcrypt registered [ 420.429710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 420.435817] alg: No test for adler32 (adler32-zlib) [ 421.236788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 421.245806] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 424.350379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 424.609537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e network_timeout -r 1 [ 424.622928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e network_timeout -r 1 [ 424.636978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 424.644332] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 424.772982] LNet: 56273:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 424.797919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 424.809252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 424.904774] LNet: 56307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.907980] LNetError: 56307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 424.915414] LNet: Removed LNI 192.168.202.4@tcp [ 425.095336] Key type .llcrypt unregistered [ 425.097060] Key type ._llcrypt unregistered [ 430.569453] Key type ._llcrypt registered [ 430.571342] Key type .llcrypt registered [ 430.636381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 430.643853] alg: No test for adler32 (adler32-zlib) [ 431.451696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 431.464149] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 434.425292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 437.100675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.114371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 437.122261] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 437.559912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e network_timeout -r 1 [ 437.574831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e network_timeout -r 1 [ 437.588887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e network_timeout -r 1 [ 437.599610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e network_timeout -r 1 [ 437.610306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e network_timeout -r 1 [ 437.623686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e network_timeout -r 1 [ 437.634698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e network_timeout -r 1 [ 437.647456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e network_timeout -r 1 [ 437.662290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 437.671764] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 437.777959] LNet: 57164:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 437.798894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 437.808578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 439.899542] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 439.985560] LNet: 57233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.987525] LNetError: 57233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.989847] LNet: Removed LNI 192.168.202.4@tcp [ 439.994329] LNet: Removed LNI 192.168.202.4@tcp1 [ 440.127460] Key type .llcrypt unregistered [ 440.128423] Key type ._llcrypt unregistered [ 445.106558] Key type ._llcrypt registered [ 445.107601] Key type .llcrypt registered [ 445.690398] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:13:27 (1760138007) [ 445.874152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 445.880955] alg: No test for adler32 (adler32-zlib) [ 446.696197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.713335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 446.721460] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 447.025290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 447.033246] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 447.472994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 447.514944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 447.542403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 447.560258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 447.580474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 447.598662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 447.616511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.4@tcp [ 447.626716] LNetError: 58321:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.202.4@tcp to recovery [ 447.711356] LNet: 1 local NIs in recovery (showing 1): 192.168.202.4@tcp [ 447.711472] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 449.759255] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 450.837715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 453.855435] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 454.026626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 454.043316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 454.070526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.077963] LNet: 58429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.080186] LNetError: 58429:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.082209] LNet: Removed LNI 192.168.202.4@tcp [ 454.090375] LNet: Removed LNI 192.168.202.4@tcp1 [ 454.099265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.118896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 454.126972] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 454.385301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 454.396287] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 454.653302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 454.665428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 454.684634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 454.699916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 454.711205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 454.722758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 454.733053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 454.743957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.4@tcp [ 454.750396] LNetError: 58695:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.202.4@tcp to recovery [ 457.183331] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 457.185133] LNet: Skipped 1 previous similar message [ 457.910360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 461.279354] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 466.254518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 466.268853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 466.283403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 468.614150] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:13:50 (1760138030) [ 468.644385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.654788] LNet: 59416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.656989] LNetError: 59416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 468.659180] LNetError: 59416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 468.661321] LNet: Removed LNI 192.168.202.4@tcp [ 468.666616] LNet: Removed LNI 192.168.202.4@tcp1 [ 468.678048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.696503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.703944] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 468.937550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 468.944297] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 469.209541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 469.238895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 469.250707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 469.271400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 469.291770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 469.305684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 469.318555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.4@tcp --health 0 [ 469.354429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 469.727289] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.4@tcp [ 470.751365] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 470.753626] LNet: Skipped 1 previous similar message [ 484.761946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 484.810368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 484.820220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 484.829523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.4@tcp --health 500 [ 484.861805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 491.075707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 491.095220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.106744] LNet: 59915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.110550] LNetError: 59915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.114434] LNetError: 59915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 491.118520] LNet: Removed LNI 192.168.202.4@tcp [ 491.127281] LNet: Removed LNI 192.168.202.4@tcp1 [ 491.133929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.154122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 491.163610] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 491.388647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 491.396710] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 491.714237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 491.727798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 491.757464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 491.772611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 491.788580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 491.808283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 491.826323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 491.837941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.4@tcp --health 0 [ 492.191384] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 492.193576] LNet: Skipped 3 previous similar messages [ 492.992611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 503.515421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 503.525749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 503.537228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 506.129208] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:14:27 (1760138067) [ 506.283935] LNet: 60965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 506.286176] LNetError: 60965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 506.288988] LNetError: 60965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 506.291046] LNet: Removed LNI 192.168.202.4@tcp [ 506.296863] LNet: Removed LNI 192.168.202.4@tcp1 [ 506.545762] Key type .llcrypt unregistered [ 506.547024] Key type ._llcrypt unregistered [ 509.028116] Key type ._llcrypt registered [ 509.029017] Key type .llcrypt registered [ 509.085112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 509.090624] alg: No test for adler32 (adler32-zlib) [ 509.894247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 509.905079] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 513.700801] Lustre: DEBUG MARKER: Initial discovery [ 513.726959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.104@tcp [ 514.816411] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 514.842772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 514.868309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.104@tcp [ 514.882473] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 514.893335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 514.910034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.104@tcp -r 1 -m GET -l 3 [ 514.923434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.104@tcp -r 1 -m GET -e local_error [ 514.937309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.104@tcp -r 1 -m PUT -l 6 [ 515.458446] Lustre: DEBUG MARKER: Force oleg204-client.virtnet to discover 192.168.202.104@tcp (in background) [ 515.481449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.104@tcp [ 516.094275] Lustre: DEBUG MARKER: Force oleg204-server to discover 192.168.202.4@tcp [ 516.847788] Lustre: DEBUG MARKER: Wait for 61713 [ 518.559474] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 519.141290] Lustre: DEBUG MARKER: Finished wait on 61713 [ 519.162081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 519.179560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 519.285373] LNet: 62169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.287591] LNetError: 62169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 519.295454] LNet: Removed LNI 192.168.202.4@tcp [ 519.476382] Key type .llcrypt unregistered [ 519.477445] Key type ._llcrypt unregistered [ 525.297648] Key type ._llcrypt registered [ 525.298766] Key type .llcrypt registered [ 525.977326] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:14:47 (1760138087) [ 526.174412] Key type .llcrypt unregistered [ 526.175832] Key type ._llcrypt unregistered [ 528.087376] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 528.149974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 528.151747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 528.153193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 530.441457] Key type ._llcrypt registered [ 530.443275] Key type .llcrypt registered [ 530.535700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 530.543192] alg: No test for adler32 (adler32-zlib) [ 531.363808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.390638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 531.399493] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 531.800882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 531.814783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 535.235725] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:14:56 (1760138096) [ 535.282353] LNet: 64224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.286433] LNetError: 64224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 535.290196] LNet: Removed LNI 192.168.202.4@tcp [ 535.298917] LNet: Removed LNI 10.1.2.3@tcp [ 535.614443] Key type .llcrypt unregistered [ 535.615568] Key type ._llcrypt unregistered [ 537.914481] Key type ._llcrypt registered [ 537.915618] Key type .llcrypt registered [ 537.978900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 537.984941] alg: No test for adler32 (adler32-zlib) [ 538.826055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.859470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 538.875307] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 539.306600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 539.319822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 539.856610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.4@tcp [ 540.896219] LNetError: Unexpected error -22 connecting to 192.168.202.4@tcp at host 192.168.202.4:988 [ 542.836863] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:15:04 (1760138104) [ 542.897780] LNet: 65344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.902389] LNetError: 65344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.907483] LNet: Removed LNI 192.168.202.4@tcp [ 542.917500] LNet: Removed LNI 10.1.2.3@tcp [ 543.286751] Key type .llcrypt unregistered [ 543.289223] Key type ._llcrypt unregistered [ 545.602402] Key type ._llcrypt registered [ 545.604250] Key type .llcrypt registered [ 545.678586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 545.685808] alg: No test for adler32 (adler32-zlib) [ 546.521743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.551115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 546.566289] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 546.983612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 546.998195] LNet: Added LNI 192.168.202.4@tcp2 [8/256/0/180] [ 547.490741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.4@tcp1 --nid 192.168.202.4@tcp2 [ 548.859660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp1 192.168.202.4@tcp2 [ 548.866739] LNetError: 66175:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.4@tcp1 to 192.168.202.4@tcp2 - Route table may be misconfigured [ 548.873857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp2 192.168.202.4@tcp1 [ 551.766781] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:15:13 (1760138113) [ 551.859632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.874977] LNet: 66748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.878607] LNetError: 66748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.881101] LNet: Removed LNI 192.168.202.4@tcp1 [ 552.934178] LNet: Removed LNI 192.168.202.4@tcp2 [ 552.953981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.990349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 553.006593] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 553.422067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 553.437054] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 553.930423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 553.955530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.4@tcp [ 553.978797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp1 192.168.202.4@tcp1 [ 554.000988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.4@tcp -d 192.168.202.4@tcp -e network_timeout [ 554.022863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -e network_timeout [ 554.046376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -e network_timeout [ 554.070469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -e network_timeout [ 554.095297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 554.108482] LNet: There was an unexpected network error while writing to 192.168.202.4: rc = -22 [ 554.120363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 554.168284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 554.219487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 555.039220] LNet: 1 local NIs in recovery (showing 1): 192.168.202.4@tcp [ 556.730829] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:15:18 (1760138118) [ 556.776792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.787354] LNet: 67598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.791215] LNetError: 67598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.794614] LNetError: 67598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 556.797493] LNet: Removed LNI 192.168.202.4@tcp [ 557.861918] LNet: Removed LNI 192.168.202.4@tcp1 [ 557.872193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.914955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 558.023372] LNet: 67639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.170663] Key type .llcrypt unregistered [ 558.172055] Key type ._llcrypt unregistered [ 562.296841] Key type ._llcrypt registered [ 562.297940] Key type .llcrypt registered [ 562.856871] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:15:24 (1760138124) [ 563.028595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 563.035031] alg: No test for adler32 (adler32-zlib) [ 563.837304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.424903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 565.133714] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:15:26 (1760138126) [ 565.194462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.212231] LNet: 68728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.226742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.258611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 565.270922] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 565.622273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 565.631719] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 565.997849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 566.012929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp1 [ 566.026014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp1 [ 568.624421] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:15:30 (1760138130) [ 569.416631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg204-server [ 570.028270] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:15:31 (1760138131) [ 570.743976] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg204-server [ 571.419837] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:15:32 (1760138132) [ 572.226745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg204-server [ 573.099455] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:15:34 (1760138134) [ 573.715596] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg204-server [ 574.408073] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:15:35 (1760138135) [ 575.253941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg204-server [ 576.038922] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:15:37 (1760138137) [ 576.738124] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg204-server [ 577.317811] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:15:38 (1760138138) [ 577.899598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg204-server [ 578.435760] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:15:40 (1760138140) [ 579.046749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg204-server [ 579.632469] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:15:41 (1760138141) [ 580.276316] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg204-server [ 580.929110] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:15:42 (1760138142) [ 581.021484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.034977] LNet: 72598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.039099] LNetError: 72598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.042556] LNet: Removed LNI 192.168.202.4@tcp [ 582.054788] LNet: Removed LNI 192.168.202.4@tcp1 [ 582.072515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.112201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 582.131268] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 582.560844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 582.611845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 582.705375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.715974] LNet: 72742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.719473] LNetError: 72742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.722355] LNetError: 72742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 582.730683] LNet: Removed LNI 192.168.202.4@tcp [ 582.744384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.776963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 582.794719] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 583.235665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 583.325893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.357653] LNet: Removed LNI 192.168.202.4@tcp [ 584.371115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.404226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 584.419910] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 584.739698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 584.771489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 584.847646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.856569] LNet: 73029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.858362] LNet: 73029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 584.860345] LNetError: 73029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.862077] LNetError: 73029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 584.869517] LNet: Removed LNI 192.168.202.4@tcp [ 584.877881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.899876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 584.909355] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 585.204047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 585.273055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.341635] LNet: Removed LNI 192.168.202.4@tcp [ 586.350892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.371785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.380189] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 586.608650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 586.642766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 586.723788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.749515] LNet: Removed LNI 192.168.202.4@tcp [ 587.759755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.778643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 587.788321] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 588.062352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 588.136415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.143705] LNetError: 73457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.146883] LNetError: 73457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 588.155410] LNet: Removed LNI 192.168.202.4@tcp [ 588.163142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.183864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 588.193402] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 588.454825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 588.489795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 588.579304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.605628] LNet: Removed LNI 192.168.202.4@tcp [ 589.619321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.653605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 589.670381] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 590.071922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 590.160781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.168612] LNet: 73743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.170733] LNet: 73743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 591.205375] LNet: Removed LNI 192.168.202.4@tcp [ 591.214570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.236541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.247842] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 591.516974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 591.551982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 591.630302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.677324] LNet: Removed LNI 192.168.202.4@tcp [ 592.685092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.705063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 592.715271] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 592.961687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 594.096048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.105595] LNetError: 74040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.108969] LNetError: 74040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 594.123446] LNet: Removed LNI 192.168.202.4@tcp [ 594.131249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.151307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.159126] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 594.415282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 594.445479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 594.530915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.557443] LNet: Removed LNI 192.168.202.4@tcp [ 595.566324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.588586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 595.601584] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 595.882637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 595.978251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.029489] LNet: Removed LNI 192.168.202.4@tcp [ 597.038811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.061235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.071533] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 597.314395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 597.336116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.348355] LNet: Removed LNI 192.168.202.4@tcp [ 597.355368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.375323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.384154] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 597.624262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 600.278324] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:16:01 (1760138161) [ 600.338861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.346801] LNet: 75156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.348799] LNet: 75156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 600.359492] LNet: Removed LNI 192.168.202.4@tcp [ 600.367492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.387029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 600.396165] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/180] [ 600.834409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.851462] LNet: Removed LNI 192.168.202.4@tcp231 [ 600.858773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.880724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 600.889529] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/180] [ 600.892262] LNet: 75299:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 601.024533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 601.042441] LNet: Removed LNI 192.168.202.4@tcp231 [ 601.053470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 601.070973] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/180] [ 601.555114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.573479] LNet: Removed LNI 192.168.202.4@tcp231 [ 601.583112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.623250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 601.639466] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/0] [ 601.643707] LNet: 75448:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 601.788647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 601.807410] LNet: Removed LNI 192.168.202.4@tcp231 [ 601.813538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 601.821781] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/0] [ 602.213485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.222505] LNetError: 75588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.225071] LNetError: 75588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 602.238329] LNet: Removed LNI 192.168.202.4@tcp231 [ 602.246479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.273762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 602.284842] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/60] [ 602.407195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 602.429468] LNet: Removed LNI 192.168.202.4@tcp231 [ 602.437324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 602.449058] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/60] [ 602.844148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.861360] LNet: Removed LNI 192.168.202.4@tcp231 [ 602.869513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.895622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 602.905443] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/180] [ 602.907538] LNet: 75744:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 602.910035] LNet: 75744:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 603.038161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 603.053349] LNet: Removed LNI 192.168.202.4@tcp231 [ 603.058534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 603.067482] LNet: Added LNI 192.168.202.4@tcp231 [8/256/0/180] [ 605.658409] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:16:07 (1760138167) [ 605.692224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.708729] LNet: Removed LNI 192.168.202.4@tcp231 [ 605.718880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.740215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 605.747906] LNet: Added LNI 192.168.202.4@tcp232 [8/256/0/180] [ 605.974723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 605.988068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 606.005414] LNet: Removed LNI 192.168.202.4@tcp232 [ 606.012268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 606.023034] LNet: Added LNI 192.168.202.4@tcp232 [8/256/0/180] [ 606.264540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 606.278895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 606.292308] LNet: Removed LNI 192.168.202.4@tcp232 [ 606.300609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 606.309713] LNet: Added LNI 192.168.202.4@tcp232 [8/256/0/180] [ 606.573308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 606.589695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 606.605364] LNet: Removed LNI 192.168.202.4@tcp232 [ 606.612739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 606.623816] LNet: Added LNI 192.168.202.4@tcp232 [8/256/0/180] [ 606.873308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 606.886236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 606.899604] LNet: Removed LNI 192.168.202.4@tcp232 [ 606.904543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 609.160868] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:16:10 (1760138170) [ 609.515929] Key type .llcrypt unregistered [ 609.517265] Key type ._llcrypt unregistered [ 611.361395] Key type ._llcrypt registered [ 611.362344] Key type .llcrypt registered [ 611.419982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 611.425312] alg: No test for adler32 (adler32-zlib) [ 612.229852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 612.239042] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 615.551632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 618.611610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.629487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 618.639686] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 619.185136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_interrupt -r 2 [ 619.201893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_interrupt -r 2 [ 619.219424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_interrupt -r 2 [ 619.237480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_interrupt -r 2 [ 619.256370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_interrupt -r 2 [ 619.275060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_interrupt -r 2 [ 619.296101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_interrupt -r 2 [ 619.315956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_interrupt -r 2 [ 619.336086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.104@tcp [ 619.347965] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 619.353712] LNet: 1 local NIs in recovery (showing 1): 192.168.202.4@tcp [ 619.494530] LNet: 78210:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 619.531391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 619.548607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 619.785048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_dropped -r 2 [ 619.800443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_dropped -r 2 [ 619.813881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_dropped -r 2 [ 619.827165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_dropped -r 2 [ 619.842469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_dropped -r 2 [ 619.856950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_dropped -r 2 [ 619.873326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_dropped -r 2 [ 619.888567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_dropped -r 2 [ 619.904305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.104@tcp [ 619.913961] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 620.041302] LNet: 78318:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 620.061761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 620.072190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 620.254921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_aborted -r 2 [ 620.275228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_aborted -r 2 [ 620.289711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_aborted -r 2 [ 620.303425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_aborted -r 2 [ 620.316194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_aborted -r 2 [ 620.328759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_aborted -r 2 [ 620.341321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_aborted -r 2 [ 620.355921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_aborted -r 2 [ 620.370199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.104@tcp [ 620.525582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 620.536069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 620.729910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_no_route -r 2 [ 620.747161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_no_route -r 2 [ 620.764839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_no_route -r 2 [ 620.776846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_no_route -r 2 [ 620.791667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_no_route -r 2 [ 620.802861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_no_route -r 2 [ 620.814688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_no_route -r 2 [ 620.825647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_no_route -r 2 [ 620.838127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.104@tcp [ 620.976163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 620.989551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 621.249947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e local_timeout -r 2 [ 621.263177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e local_timeout -r 2 [ 621.277486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e local_timeout -r 2 [ 621.294553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e local_timeout -r 2 [ 621.315920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e local_timeout -r 2 [ 621.329845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e local_timeout -r 2 [ 621.346325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e local_timeout -r 2 [ 621.360960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e local_timeout -r 2 [ 621.377408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.104@tcp [ 621.385579] LNet: There was an unexpected network error while writing to 192.168.202.104: rc = -22 [ 621.387603] LNet: Skipped 3 previous similar messages [ 621.523981] LNet: 78642:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 621.528267] LNet: 78642:0:(api-ni.c:339:recovery_interval_set()) Skipped 2 previous similar messages [ 621.554295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 621.566278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 621.771200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp -m GET -e remote_dropped -r 2 [ 621.783766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.104@tcp1 -m GET -e remote_dropped -r 2 [ 621.795297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp -m GET -e remote_dropped -r 2 [ 621.807918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp -d 192.168.202.4@tcp1 -m GET -e remote_dropped -r 2 [ 621.826322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp -m GET -e remote_dropped -r 2 [ 621.840184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.104@tcp1 -m GET -e remote_dropped -r 2 [ 621.852285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp -m GET -e remote_dropped -r 2 [ 621.867749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.4@tcp1 -d 192.168.202.4@tcp1 -m GET -e remote_dropped -r 2 [ 621.883576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.4@tcp 192.168.202.104@tcp [ 621.895498] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.104@tcp [ 622.028317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 622.038938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 624.237286] LNet: 78818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.240415] LNetError: 78818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.243318] LNet: Removed LNI 192.168.202.4@tcp [ 624.252341] LNet: Removed LNI 192.168.202.4@tcp1 [ 624.424640] Key type .llcrypt unregistered [ 624.425838] Key type ._llcrypt unregistered [ 631.000397] Key type ._llcrypt registered [ 631.001678] Key type .llcrypt registered [ 631.690205] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:16:33 (1760138193) [ 631.874460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 631.880259] alg: No test for adler32 (adler32-zlib) [ 632.699963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.723134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.734001] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 633.125219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 633.164453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 633.195765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 633.243591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 633.278169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 633.306760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 636.557529] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:16:38 (1760138198) [ 636.612435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.628711] LNet: 80364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.634205] LNetError: 80364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.644512] LNet: Removed LNI 192.168.202.4@tcp [ 636.658744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.693316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 636.704874] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 639.699200] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:16:41 (1760138201) [ 639.791380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.798628] LNet: 81070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.801009] LNetError: 81070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.807910] LNet: Removed LNI 192.168.202.4@tcp [ 639.816566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.839085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.850369] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 643.106348] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:16:44 (1760138204) [ 643.822664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 644.642581] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:16:46 (1760138206) [ 644.839361] LNet: 82102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.843382] LNetError: 82102:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.854842] LNet: Removed LNI 192.168.202.4@tcp [ 645.186668] Key type .llcrypt unregistered [ 645.188884] Key type ._llcrypt unregistered [ 647.589393] Key type ._llcrypt registered [ 647.591221] Key type .llcrypt registered [ 647.663598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.672249] alg: No test for adler32 (adler32-zlib) [ 648.475742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 648.484552] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 651.797619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 653.971472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.104@tcp [ 654.061453] LNet: 82512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.063315] LNetError: 82512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.075277] LNet: Removed LNI 192.168.202.4@tcp [ 654.230357] Key type .llcrypt unregistered [ 654.231362] Key type ._llcrypt unregistered [ 658.614929] Key type ._llcrypt registered [ 658.616048] Key type .llcrypt registered [ 659.211760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 659.951721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 660.822138] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:17:02 (1760138222) [ 660.992421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 660.998491] alg: No test for adler32 (adler32-zlib) [ 661.796560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.821562] LNet: 83594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.035275] Key type .llcrypt unregistered [ 662.036302] Key type ._llcrypt unregistered [ 664.119533] Key type ._llcrypt registered [ 664.120612] Key type .llcrypt registered [ 664.175581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.180081] alg: No test for adler32 (adler32-zlib) [ 664.979928] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 664.981745] LNet: 83845:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 664.989365] LNetError: 83857:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 664.995992] LNetError: 83857:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.7@tcp has gone from up to down [ 665.162563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.255142] LNetError: 83857:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.9@tcp has gone from up to down [ 668.259197] LNetError: 83857:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 668.262392] LNet: 83904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.266500] LNetError: 83904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.278625] LNet: Removed LNI 192.168.202.4@tcp [ 668.290845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.323567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 668.337533] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 668.341385] LNet: 83912:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 668.347355] LNet: 83912:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 668.352988] LNetError: 83912:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.6@tcp is a gateway. Can not delete it [ 668.359441] LNetError: 83912:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.202.6@tcp is a router. Can not be deleted [ 668.368730] LNetError: 83909:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 668.377852] LNetError: 83909:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 668.385673] LNet: 83912:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 670.964371] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:17:12 (1760138232) [ 671.135451] LNetError: 83923:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.202.8@tcp has gone from up to down [ 671.139181] LNetError: 83923:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 671.730977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg204-server [ 672.471897] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:17:13 (1760138233) [ 673.225910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg204-server [ 673.842816] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:17:15 (1760138235) [ 674.466654] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 676.119179] Key type lgssc registered [ 676.635814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 686.346392] Key type lgssc unregistered [ 686.550666] LNet: 86328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.557295] LNetError: 86328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.573275] LNet: Removed LNI 192.168.202.4@tcp [ 686.893812] Key type .llcrypt unregistered [ 686.895865] Key type ._llcrypt unregistered [ 692.842949] Key type ._llcrypt registered [ 692.844353] Key type .llcrypt registered [ 693.434704] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:17:34 (1760138254) [ 693.595152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 693.600806] alg: No test for adler32 (adler32-zlib) [ 694.423315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.461764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 694.485895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 694.508915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 694.528990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 697.377738] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:17:38 (1760138258) [ 697.427915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.440371] LNet: 87725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.451786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.483312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.493334] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 697.852293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.859816] LNetError: 87871:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.868453] LNet: Removed LNI 192.168.202.4@tcp [ 697.876084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.897088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 697.906218] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 698.046159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.053216] LNet: 87899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.054976] LNet: 87899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 698.068536] LNet: Removed LNI 192.168.202.4@tcp [ 698.077593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.101812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 698.111734] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 700.489689] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:17:42 (1760138262) [ 700.525436] LNet: 88481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.528379] LNetError: 88481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.530713] LNetError: 88481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 700.537949] LNet: Removed LNI 192.168.202.4@tcp [ 700.799644] Key type .llcrypt unregistered [ 700.800746] Key type ._llcrypt unregistered [ 702.747121] Key type ._llcrypt registered [ 702.748947] Key type .llcrypt registered [ 702.811861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.816971] alg: No test for adler32 (adler32-zlib) [ 703.636686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.674391] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 703.676682] LNet: *** cfs_fail_loc=e003, val=100*** [ 703.922935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 703.930075] Ping reply received with mapping: [ 703.930970] found metadata reply NI[192.168.202.4@tcp] [ 703.932131] 04 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 703.933220] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 703.934321] 5a 5a 5a 5a 5a 5a 5a 5a [ 703.975793] LNet: 88878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.977960] LNetError: 88878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.984331] LNet: Removed LNI 192.168.202.4@tcp [ 704.208263] Key type .llcrypt unregistered [ 704.209312] Key type ._llcrypt unregistered [ 708.232131] Key type ._llcrypt registered [ 708.233436] Key type .llcrypt registered [ 708.793343] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:17:50 (1760138270) [ 709.005175] Key type .llcrypt unregistered [ 709.006143] Key type ._llcrypt unregistered [ 711.026854] Key type ._llcrypt registered [ 711.028233] Key type .llcrypt registered [ 711.082635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.087849] alg: No test for adler32 (adler32-zlib) [ 711.960296] LNetError: 89913:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 712.114540] Key type .llcrypt unregistered [ 712.116025] Key type ._llcrypt unregistered [ 714.359577] Key type ._llcrypt registered [ 714.360776] Key type .llcrypt registered [ 714.990778] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:17:56 (1760138276) [ 715.201490] Key type .llcrypt unregistered [ 715.203259] Key type ._llcrypt unregistered [ 717.248953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 717.250850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 719.452938] Key type ._llcrypt registered [ 719.454127] Key type .llcrypt registered [ 719.524385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 719.531033] alg: No test for adler32 (adler32-zlib) [ 720.333981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.354812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 724.934549] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:18:06 (1760138286) [ 724.998432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.017529] LNet: 91677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.023743] LNetError: 91677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 725.035131] LNet: Removed LNI 10.1.2.3@tcp [ 725.052731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.477914] Key type .llcrypt unregistered [ 725.479965] Key type ._llcrypt unregistered [ 727.538749] Key type ._llcrypt registered [ 727.539834] Key type .llcrypt registered [ 727.593696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.601356] alg: No test for adler32 (adler32-zlib) [ 728.404181] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 728.427304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 728.485583] LNet: 91964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.488332] LNetError: 91964:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.503414] LNet: Removed LNI 192.168.202.4@tcp [ 728.807538] Key type .llcrypt unregistered [ 728.809092] Key type ._llcrypt unregistered [ 732.879686] Key type ._llcrypt registered [ 732.881075] Key type .llcrypt registered [ 733.517459] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:18:14 (1760138294) [ 733.791340] Key type .llcrypt unregistered [ 733.793292] Key type ._llcrypt unregistered [ 736.184422] Key type ._llcrypt registered [ 736.185465] Key type .llcrypt registered [ 736.246345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 736.251162] alg: No test for adler32 (adler32-zlib) [ 739.504236] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:18:20 (1760138300) [ 739.571869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.595835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.622925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.634810] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 739.902553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.917181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 739.936740] LNet: Removed LNI 192.168.202.4@tcp [ 740.040763] LNet: 93694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.234533] Key type .llcrypt unregistered [ 740.236190] Key type ._llcrypt unregistered [ 744.631572] Key type ._llcrypt registered [ 744.632481] Key type .llcrypt registered [ 745.185503] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:18:26 (1760138306) [ 745.334736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.339241] alg: No test for adler32 (adler32-zlib) [ 746.143077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.161596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.171172] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 746.529386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 746.549690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.4@tcp [ 746.560610] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 746.563261] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 746.565998] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 746, 0, 0 [ 746.569352] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 746.571162] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 746.573681] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 746.575776] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 746, 0, 0 [ 746.578653] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 746.580237] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 746.582218] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 746.583815] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 746, 0, 796 [ 746.586370] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 746.587707] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 746.589437] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 746.590928] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 746, 0, 796 [ 746.593080] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 749.471757] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:18:30 (1760138310) [ 749.649546] LNet: 95228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.652510] LNetError: 95228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.659819] LNet: Removed LNI 192.168.202.4@tcp [ 749.913356] Key type .llcrypt unregistered [ 749.915185] Key type ._llcrypt unregistered [ 752.042198] Key type ._llcrypt registered [ 752.043141] Key type .llcrypt registered [ 752.108520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.113637] alg: No test for adler32 (adler32-zlib) [ 752.949585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 752.966971] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 756.742323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 760.161158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.176692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 760.187386] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 760.725802] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 760.780225] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.782165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.783728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.985220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 762.998774] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 768.438314] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:18:49 (1760138329) [ 768.504495] LNet: 96718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.508434] LNetError: 96718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.513392] LNet: Removed LNI 192.168.202.4@tcp [ 769.567133] LNet: Removed LNI 192.168.202.4@tcp1 [ 769.568404] LNetError: 96718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 769.570464] LNetError: 96718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 769.578673] LNet: Removed LNI 10.1.2.3@tcp99 [ 769.905477] Key type .llcrypt unregistered [ 769.907030] Key type ._llcrypt unregistered [ 772.402705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.404039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.647259] Key type ._llcrypt registered [ 774.649102] Key type .llcrypt registered [ 774.753566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.761156] alg: No test for adler32 (adler32-zlib) [ 775.582635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.608331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.622157] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 775.915117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 775.925261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 776.804916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 776.835681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 776.858169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 776.868790] LNetError: 97456:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 776.887745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 776.913488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 777.099040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 777.127512] LNet: 97476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.129740] LNetError: 97476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.132022] LNet: Removed LNI 192.168.202.4@tcp [ 777.140501] LNet: Removed LNI 10.1.2.3@tcp [ 777.444403] Key type .llcrypt unregistered [ 777.445933] Key type ._llcrypt unregistered [ 781.443128] Key type ._llcrypt registered [ 781.444542] Key type .llcrypt registered [ 782.113990] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:19:03 (1760138343) [ 782.264520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.269930] alg: No test for adler32 (adler32-zlib) [ 783.095301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.119921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 783.131903] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 783.379518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.4@tcp [ 783.405374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg204-client@tcp [ 786.375848] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:19:07 (1760138347) [ 786.430727] LNet: 98998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.435864] LNetError: 98998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.447825] LNet: Removed LNI 192.168.202.4@tcp [ 786.784788] Key type .llcrypt unregistered [ 786.786892] Key type ._llcrypt unregistered [ 789.234688] Key type ._llcrypt registered [ 789.236491] Key type .llcrypt registered [ 789.285222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.290255] alg: No test for adler32 (adler32-zlib) [ 790.099770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 790.109160] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 790.126829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 793.035505] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:19:14 (1760138354) [ 793.743794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 794.591203] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:19:16 (1760138356) [ 795.371369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 796.260983] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:19:17 (1760138357) [ 796.320167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.335086] LNet: 100416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.339158] LNetError: 100416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.349756] LNet: Removed LNI 192.168.202.4@tcp [ 796.371112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.402295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.413036] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 796.817256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 796.849281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 799.670713] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:19:21 (1760138361) [ 799.741093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.753041] LNet: 101124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.756158] LNetError: 101124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.767564] LNet: Removed LNI 192.168.202.4@tcp [ 799.774376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.796707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 799.810945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 800.142325] Key type .llcrypt unregistered [ 800.144301] Key type ._llcrypt unregistered [ 804.981684] Key type ._llcrypt registered [ 804.983114] Key type .llcrypt registered [ 805.580859] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:19:27 (1760138367) [ 805.781792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.789202] alg: No test for adler32 (adler32-zlib) [ 806.603304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.625938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.639307] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 806.887131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 806.910159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 809.259524] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:19:30 (1760138370) [ 809.294956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.304723] LNet: 102669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.306986] LNetError: 102669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.315424] LNet: Removed LNI 192.168.202.4@tcp [ 809.323729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.341384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 809.352267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 811.479215] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 811.989346] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:19:33 (1760138373) [ 812.022156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.032405] LNet: 103389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.043955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.130221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.137972] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 812.359562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 812.368472] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.658155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.4@tcp [ 812.675992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 812.704563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.4@tcp -m PUT -e local_timeout -r 1 [ 812.734047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 812.760497] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.764226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.868616] LNetError: 103423:0:(lib-move.c:4947:lnet_parse()) 192.168.202.4@tcp, src 192.168.202.4@tcp: Dropping PUT (error -108 looking up sender) [ 813.919593] LNet: 103699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.924531] LNetError: 103699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.929238] LNet: Removed LNI 192.168.202.4@tcp [ 813.937611] LNet: Removed LNI 10.1.2.3@tcp [ 814.106462] Key type .llcrypt unregistered [ 814.107636] Key type ._llcrypt unregistered [ 818.313381] Key type ._llcrypt registered [ 818.314783] Key type .llcrypt registered [ 818.883923] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:19:40 (1760138380) [ 819.047886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 819.053926] alg: No test for adler32 (adler32-zlib) [ 819.862978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.003742] LNet: 104547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.283383] Key type .llcrypt unregistered [ 820.285189] Key type ._llcrypt unregistered [ 822.584421] Key type ._llcrypt registered [ 822.585465] Key type .llcrypt registered [ 822.642502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.647205] alg: No test for adler32 (adler32-zlib) [ 823.450489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 823.459833] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 826.986273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 827.073923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 827.406974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.104@tcp [ 827.531409] LNet: 104983:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 827.551102] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.104@tcp [ 827.555993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 827.569914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 827.868717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 828.029810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 828.040908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 828.282504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.104@tcp [ 828.425138] LNet: 105193:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 828.427426] LNet: 105193:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 828.452098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 828.463877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 828.739955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 828.890290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 828.901706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 829.027817] LNet: 105339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.033180] LNetError: 105339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.043597] LNet: Removed LNI 192.168.202.4@tcp [ 829.270730] Key type .llcrypt unregistered [ 829.272991] Key type ._llcrypt unregistered [ 835.527647] Key type ._llcrypt registered [ 835.529219] Key type .llcrypt registered [ 836.062233] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:19:57 (1760138397) [ 836.206092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.211136] alg: No test for adler32 (adler32-zlib) [ 837.051637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.284263] LNet: 106221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.700533] Key type .llcrypt unregistered [ 837.702655] Key type ._llcrypt unregistered [ 839.950800] Key type ._llcrypt registered [ 839.952492] Key type .llcrypt registered [ 840.020402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.025379] alg: No test for adler32 (adler32-zlib) [ 840.859226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 840.870783] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 844.414801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.104@tcp [ 847.386911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.398877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 847.406321] LNet: Added LNI 192.168.202.4@tcp1 [8/256/0/180] [ 847.684696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 847.895901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.104@tcp [ 848.011821] LNet: 106857:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 848.031240] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.104@tcp [ 848.036174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 848.052507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 848.299385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 848.417189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 848.427723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 848.618943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.104@tcp [ 848.716845] LNet: 107067:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 848.719173] LNet: 107067:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 848.740459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 848.751575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 848.940684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 849.060454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 849.072063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 851.405697] LNet: 107247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.407889] LNetError: 107247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.410859] LNet: Removed LNI 192.168.202.4@tcp [ 851.419423] LNet: Removed LNI 192.168.202.4@tcp1 [ 851.627611] Key type .llcrypt unregistered [ 851.628754] Key type ._llcrypt unregistered [ 858.003197] Key type ._llcrypt registered [ 858.004307] Key type .llcrypt registered [ 858.426263] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 746 sec =========== 19:20:20 (1760138420) [ 858.673293] Key type .llcrypt unregistered [ 858.674639] Key type ._llcrypt unregistered [ 863.802580] Key type ._llcrypt registered [ 863.803831] Key type .llcrypt registered [ 863.965061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.969625] alg: No test for adler32 (adler32-zlib) [ 864.844087] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 864.949560] LNet: Added LNI 192.168.202.4@tcp [8/256/0/180] [ 866.543200] Key type lgssc registered [ 867.119354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 893.776356] Lustre: Mounted lustre-client [ 896.588208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 904.429263] LustreError: 110460:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9e8c493f2000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 904.433769] LustreError: 110460:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 904.443993] Lustre: Unmounted lustre-client [ 931.305821] Key type lgssc unregistered [ 931.423470] LNet: 110944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.425577] LNetError: 110944:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.433793] LNet: Removed LNI 192.168.202.4@tcp [ 931.673098] Key type .llcrypt unregistered [ 931.674265] Key type ._llcrypt unregistered